{"id":1632,"date":"2023-11-13T19:19:18","date_gmt":"2023-11-13T19:19:18","guid":{"rendered":"https:\/\/www.ethanepperly.com\/?p=1632"},"modified":"2023-11-24T02:53:41","modified_gmt":"2023-11-24T02:53:41","slug":"does-sketching-work","status":"publish","type":"post","link":"https:\/\/www.ethanepperly.com\/index.php\/2023\/11\/13\/does-sketching-work\/","title":{"rendered":"Does Sketching Work?"},"content":{"rendered":"\n<p>I&#8217;m excited to share that my paper, <em><a href=\"https:\/\/arxiv.org\/abs\/2311.04362\">Fast and forward stable randomized algorithms for linear least-squares problems<\/a><\/em> has been released as a preprint on arXiv.<\/p>\n\n\n\n<p>With the release of this paper, now seemed like a great time to discuss a topic I\u2019ve been wanting to write about for a while: <a href=\"https:\/\/ar5iv.labs.arxiv.org\/html\/2002.01387#S8\"><em>sketching<\/em><\/a>. For the past two decades, sketching has become a widely used algorithmic tool in matrix computations. Despite this long history, questions still seem to be lingering about whether sketching really works:<\/p>\n\n\n\n<figure class=\"wp-block-embed is-type-rich is-provider-twitter wp-block-embed-twitter\"><div class=\"wp-block-embed__wrapper\">\n<blockquote class=\"twitter-tweet\" data-width=\"550\" data-dnt=\"true\"><p lang=\"en\" dir=\"ltr\">does sketching actually work in practice? i want to know why it&#39;s not really used in deep learning<\/p>&mdash; typedfemale (@typedfemale) <a href=\"https:\/\/twitter.com\/typedfemale\/status\/1572838445499973635?ref_src=twsrc%5Etfw\">September 22, 2022<\/a><\/blockquote><script async src=\"https:\/\/platform.twitter.com\/widgets.js\" charset=\"utf-8\"><\/script>\n<\/div><\/figure>\n\n\n\n<p>In this post, I want to take a critical look at the question &#8220;does sketching work&#8221;? Answering this question requires answering two basic questions:<\/p>\n\n\n\n<ol class=\"wp-block-list\"><li>What is sketching?<\/li><li>What would it mean for sketching to work?<\/li><\/ol>\n\n\n\n<p>I think a large part of the disagreement over the efficacy of sketching boils down to different answers to these questions. By considering different possible answers to these questions, I hope to provide a balanced perspective on the utility of sketching as an algorithmic primitive for solving linear algebra problems.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Sketching<\/h2>\n\n\n\n<p>In matrix computations, <em>sketching<\/em> is really a synonym for (linear) <em><a href=\"https:\/\/en.wikipedia.org\/wiki\/Dimensionality_reduction\">dimensionality reduction<\/a><\/em>. Suppose we are solving a problem involving one or more high-dimensional vectors <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-687661d959fd9cfac193c411ee6bc2a7_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#98;&#32;&#92;&#105;&#110;&#32;&#92;&#114;&#101;&#97;&#108;&#94;&#110;\" title=\"Rendered by QuickLaTeX.com\" height=\"13\" width=\"50\" style=\"vertical-align: -1px;\"\/> or perhaps a tall matrix <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-3c615dd3d0ecf20fa079cef2a3b847f6_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#65;&#92;&#105;&#110;&#32;&#92;&#114;&#101;&#97;&#108;&#94;&#123;&#110;&#92;&#116;&#105;&#109;&#101;&#115;&#32;&#107;&#125;\" title=\"Rendered by QuickLaTeX.com\" height=\"16\" width=\"74\" style=\"vertical-align: -1px;\"\/>. A sketching matrix is a <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-91cf6c6859269c70836c11cac60fc0c2_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#100;&#92;&#116;&#105;&#109;&#101;&#115;&#32;&#110;\" title=\"Rendered by QuickLaTeX.com\" height=\"12\" width=\"42\" style=\"vertical-align: 0px;\"\/> matrix <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-9169a11ed0c9a455fe0958b71630faf6_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#83;&#32;&#92;&#105;&#110;&#32;&#92;&#114;&#101;&#97;&#108;&#94;&#123;&#100;&#92;&#116;&#105;&#109;&#101;&#115;&#32;&#110;&#125;\" title=\"Rendered by QuickLaTeX.com\" height=\"16\" width=\"73\" style=\"vertical-align: -1px;\"\/> where <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-09b5c91b060939f88251d5fc1023dba7_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#100;&#32;&#92;&#108;&#108;&#32;&#110;\" title=\"Rendered by QuickLaTeX.com\" height=\"13\" width=\"48\" style=\"vertical-align: -1px;\"\/>. When multiplied into a high-dimensional vector <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-3ab6f6c64ce17f786a81f8cc8fdcec90_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#98;\" title=\"Rendered by QuickLaTeX.com\" height=\"12\" width=\"8\" style=\"vertical-align: 0px;\"\/> or tall matrix <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-11f4f587954b361e7d78940f65b8d70d_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#65;\" title=\"Rendered by QuickLaTeX.com\" height=\"13\" width=\"13\" style=\"vertical-align: 0px;\"\/>, the sketching matrix <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-b89471a11dd7fd85184a38ddb3ea9145_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#83;\" title=\"Rendered by QuickLaTeX.com\" height=\"12\" width=\"12\" style=\"vertical-align: 0px;\"\/> produces compressed or &#8220;sketched&#8221; versions <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-bd76151778055d405aee0532cc3a8f23_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#83;&#98;\" title=\"Rendered by QuickLaTeX.com\" height=\"12\" width=\"20\" style=\"vertical-align: 0px;\"\/> and <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-f6b6ffdfa7888fa001ee9e26544c7384_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#83;&#65;\" title=\"Rendered by QuickLaTeX.com\" height=\"13\" width=\"25\" style=\"vertical-align: 0px;\"\/> that are much smaller than the original vector <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-3ab6f6c64ce17f786a81f8cc8fdcec90_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#98;\" title=\"Rendered by QuickLaTeX.com\" height=\"12\" width=\"8\" style=\"vertical-align: 0px;\"\/> and matrix <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-11f4f587954b361e7d78940f65b8d70d_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#65;\" title=\"Rendered by QuickLaTeX.com\" height=\"13\" width=\"13\" style=\"vertical-align: 0px;\"\/>.<\/p>\n\n\n<div class=\"wp-block-image\">\n<figure class=\"aligncenter size-large is-resized\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/uploads\/2023\/11\/sketching-1024x525.png\" alt=\"\" class=\"wp-image-1634\" width=\"768\" height=\"394\" srcset=\"https:\/\/www.ethanepperly.com\/wp-content\/uploads\/2023\/11\/sketching-1024x525.png 1024w, https:\/\/www.ethanepperly.com\/wp-content\/uploads\/2023\/11\/sketching-300x154.png 300w, https:\/\/www.ethanepperly.com\/wp-content\/uploads\/2023\/11\/sketching-768x394.png 768w, https:\/\/www.ethanepperly.com\/wp-content\/uploads\/2023\/11\/sketching-1536x788.png 1536w, https:\/\/www.ethanepperly.com\/wp-content\/uploads\/2023\/11\/sketching-2048x1051.png 2048w\" sizes=\"auto, (max-width: 768px) 100vw, 768px\" \/><\/figure><\/div>\n\n\n<p>Let <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-32ae15443dd7f51e8a3a03d480604464_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#92;&#109;&#97;&#116;&#104;&#115;&#102;&#123;&#69;&#125;&#61;&#92;&#123;&#120;&#95;&#49;&#44;&#92;&#108;&#100;&#111;&#116;&#115;&#44;&#120;&#95;&#112;&#92;&#125;\" title=\"Rendered by QuickLaTeX.com\" height=\"20\" width=\"125\" style=\"vertical-align: -6px;\"\/> be a collection of vectors. For <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-b89471a11dd7fd85184a38ddb3ea9145_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#83;\" title=\"Rendered by QuickLaTeX.com\" height=\"12\" width=\"12\" style=\"vertical-align: 0px;\"\/> to be a &#8220;good&#8221; sketching matrix for <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-a3e55f8dfa3cc0be5bec482e7752a94c_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#92;&#109;&#97;&#116;&#104;&#115;&#102;&#123;&#69;&#125;\" title=\"Rendered by QuickLaTeX.com\" height=\"13\" width=\"9\" style=\"vertical-align: 0px;\"\/>, we require that <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-b89471a11dd7fd85184a38ddb3ea9145_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#83;\" title=\"Rendered by QuickLaTeX.com\" height=\"12\" width=\"12\" style=\"vertical-align: 0px;\"\/> preserves the lengths of every vector in <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-a3e55f8dfa3cc0be5bec482e7752a94c_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#92;&#109;&#97;&#116;&#104;&#115;&#102;&#123;&#69;&#125;\" title=\"Rendered by QuickLaTeX.com\" height=\"13\" width=\"9\" style=\"vertical-align: 0px;\"\/> up to a distortion parameter <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-b4e8fbd2ec47658f239f615abb70724e_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#92;&#118;&#97;&#114;&#101;&#112;&#115;&#105;&#108;&#111;&#110;&#62;&#48;\" title=\"Rendered by QuickLaTeX.com\" height=\"14\" width=\"41\" style=\"vertical-align: -2px;\"\/>: <p class=\"ql-center-displayed-equation\" style=\"line-height: 19px;\"><span class=\"ql-right-eqno\"> (1) <\/span><span class=\"ql-left-eqno\"> &nbsp; <\/span><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-735df49a45ff5d0bb27d478569004bdd_l3.png\" height=\"19\" width=\"251\" class=\"ql-img-displayed-equation quicklatex-auto-format\" alt=\"&#92;&#091;&#40;&#49;&#45;&#92;&#118;&#97;&#114;&#101;&#112;&#115;&#105;&#108;&#111;&#110;&#41;&#32;&#92;&#110;&#111;&#114;&#109;&#123;&#120;&#125;&#92;&#108;&#101;&#92;&#110;&#111;&#114;&#109;&#123;&#83;&#120;&#125;&#92;&#108;&#101;&#40;&#49;&#43;&#92;&#118;&#97;&#114;&#101;&#112;&#115;&#105;&#108;&#111;&#110;&#41;&#92;&#110;&#111;&#114;&#109;&#123;&#120;&#125;&#32;&#92;&#093;\" title=\"Rendered by QuickLaTeX.com\"\/><\/p>for every <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-b312d649591164b7149ed0756f694a76_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#120;\" title=\"Rendered by QuickLaTeX.com\" height=\"8\" width=\"10\" style=\"vertical-align: 0px;\"\/> in <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-a3e55f8dfa3cc0be5bec482e7752a94c_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#92;&#109;&#97;&#116;&#104;&#115;&#102;&#123;&#69;&#125;\" title=\"Rendered by QuickLaTeX.com\" height=\"13\" width=\"9\" style=\"vertical-align: 0px;\"\/>.<\/p>\n\n\n\n<p>For linear algebra problems, we often want to sketch a matrix <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-11f4f587954b361e7d78940f65b8d70d_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#65;\" title=\"Rendered by QuickLaTeX.com\" height=\"13\" width=\"13\" style=\"vertical-align: 0px;\"\/>. In this case, the appropriate set <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-a3e55f8dfa3cc0be5bec482e7752a94c_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#92;&#109;&#97;&#116;&#104;&#115;&#102;&#123;&#69;&#125;\" title=\"Rendered by QuickLaTeX.com\" height=\"13\" width=\"9\" style=\"vertical-align: 0px;\"\/> that we want our sketch to be &#8220;good&#8221; for is the <a href=\"https:\/\/en.wikipedia.org\/wiki\/Row_and_column_spaces\">column space<\/a> of the matrix <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-11f4f587954b361e7d78940f65b8d70d_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#65;\" title=\"Rendered by QuickLaTeX.com\" height=\"13\" width=\"13\" style=\"vertical-align: 0px;\"\/>, defined to be <p class=\"ql-center-displayed-equation\" style=\"line-height: 22px;\"><span class=\"ql-right-eqno\"> &nbsp; <\/span><span class=\"ql-left-eqno\"> &nbsp; <\/span><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-5ecde6b43df29caba14fd43682b4cc86_l3.png\" height=\"22\" width=\"189\" class=\"ql-img-displayed-equation quicklatex-auto-format\" alt=\"&#92;&#091;&#92;&#111;&#112;&#101;&#114;&#97;&#116;&#111;&#114;&#110;&#97;&#109;&#101;&#123;&#99;&#111;&#108;&#125;&#40;&#65;&#41;&#32;&#92;&#99;&#111;&#108;&#111;&#110;&#101;&#113;&#113;&#32;&#92;&#123;&#32;&#65;&#120;&#32;&#58;&#32;&#120;&#32;&#92;&#105;&#110;&#32;&#92;&#114;&#101;&#97;&#108;&#94;&#107;&#32;&#92;&#125;&#46;&#92;&#093;\" title=\"Rendered by QuickLaTeX.com\"\/><\/p>Remarkably, there exist many sketching matrices that achieve distortion <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-68eb29837031f4afbd699052143a2826_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#92;&#118;&#97;&#114;&#101;&#112;&#115;&#105;&#108;&#111;&#110;\" title=\"Rendered by QuickLaTeX.com\" height=\"8\" width=\"8\" style=\"vertical-align: 0px;\"\/> for <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-daff62dfd4392d5f4bc951c8d49248f7_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#92;&#109;&#97;&#116;&#104;&#115;&#102;&#123;&#69;&#125;&#61;&#92;&#111;&#112;&#101;&#114;&#97;&#116;&#111;&#114;&#110;&#97;&#109;&#101;&#123;&#99;&#111;&#108;&#125;&#40;&#65;&#41;\" title=\"Rendered by QuickLaTeX.com\" height=\"19\" width=\"81\" style=\"vertical-align: -5px;\"\/> with an output dimension of roughly <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-135b8a27fc5a1b88a01311cdb204b84a_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#100;&#32;&#92;&#97;&#112;&#112;&#114;&#111;&#120;&#32;&#107;&#32;&#47;&#32;&#92;&#118;&#97;&#114;&#101;&#112;&#115;&#105;&#108;&#111;&#110;&#94;&#50;\" title=\"Rendered by QuickLaTeX.com\" height=\"20\" width=\"67\" style=\"vertical-align: -5px;\"\/>. In particular, the sketching dimension <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-1b031c8458e71f0e4df82ad61d36c0ca_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#100;\" title=\"Rendered by QuickLaTeX.com\" height=\"12\" width=\"10\" style=\"vertical-align: 0px;\"\/> is proportional to the number of columns <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-f65286b751f121928913d4aa91d94ee9_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#107;\" title=\"Rendered by QuickLaTeX.com\" height=\"12\" width=\"9\" style=\"vertical-align: 0px;\"\/> of <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-11f4f587954b361e7d78940f65b8d70d_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#65;\" title=\"Rendered by QuickLaTeX.com\" height=\"13\" width=\"13\" style=\"vertical-align: 0px;\"\/>. This is pretty neat! We can design a single sketching matrix <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-b89471a11dd7fd85184a38ddb3ea9145_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#83;\" title=\"Rendered by QuickLaTeX.com\" height=\"12\" width=\"12\" style=\"vertical-align: 0px;\"\/> which preserves the lengths of all infinitely-many vectors <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-3da95081ac78cee0692e52bb881df16c_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#65;&#120;\" title=\"Rendered by QuickLaTeX.com\" height=\"13\" width=\"23\" style=\"vertical-align: 0px;\"\/> in the column space of <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-11f4f587954b361e7d78940f65b8d70d_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#65;\" title=\"Rendered by QuickLaTeX.com\" height=\"13\" width=\"13\" style=\"vertical-align: 0px;\"\/>.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Sketching Matrices<\/h2>\n\n\n\n<p>There are many types of sketching matrices, each with different benefits and drawbacks. Many sketching matrices are based on <em><a href=\"https:\/\/www.ethanepperly.com\/index.php\/2021\/08\/11\/why-randomized-algorithms\/\">randomized<\/a> constructions<\/em> in the sense that entries of <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-b89471a11dd7fd85184a38ddb3ea9145_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#83;\" title=\"Rendered by QuickLaTeX.com\" height=\"12\" width=\"12\" style=\"vertical-align: 0px;\"\/> are chosen to be random numbers. Broadly, sketching matrices can be classified into two types:<\/p>\n\n\n\n<ul class=\"wp-block-list\"><li><strong>Data-dependent sketches.<\/strong> The sketching matrix <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-b89471a11dd7fd85184a38ddb3ea9145_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#83;\" title=\"Rendered by QuickLaTeX.com\" height=\"12\" width=\"12\" style=\"vertical-align: 0px;\"\/> is constructed to work for a specific set of input vectors <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-a3e55f8dfa3cc0be5bec482e7752a94c_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#92;&#109;&#97;&#116;&#104;&#115;&#102;&#123;&#69;&#125;\" title=\"Rendered by QuickLaTeX.com\" height=\"13\" width=\"9\" style=\"vertical-align: 0px;\"\/>.<\/li><li><strong>Oblivious sketches.<\/strong> The sketching matrix <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-b89471a11dd7fd85184a38ddb3ea9145_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#83;\" title=\"Rendered by QuickLaTeX.com\" height=\"12\" width=\"12\" style=\"vertical-align: 0px;\"\/> is designed to work for an arbitrary set of input vectors <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-a3e55f8dfa3cc0be5bec482e7752a94c_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#92;&#109;&#97;&#116;&#104;&#115;&#102;&#123;&#69;&#125;\" title=\"Rendered by QuickLaTeX.com\" height=\"13\" width=\"9\" style=\"vertical-align: 0px;\"\/> of a given size (i.e., <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-a3e55f8dfa3cc0be5bec482e7752a94c_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#92;&#109;&#97;&#116;&#104;&#115;&#102;&#123;&#69;&#125;\" title=\"Rendered by QuickLaTeX.com\" height=\"13\" width=\"9\" style=\"vertical-align: 0px;\"\/> has <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-726e7ac82690902493102f577788aa40_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#112;\" title=\"Rendered by QuickLaTeX.com\" height=\"12\" width=\"10\" style=\"vertical-align: -4px;\"\/> elements) or dimension (<img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-a3e55f8dfa3cc0be5bec482e7752a94c_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#92;&#109;&#97;&#116;&#104;&#115;&#102;&#123;&#69;&#125;\" title=\"Rendered by QuickLaTeX.com\" height=\"13\" width=\"9\" style=\"vertical-align: 0px;\"\/> is a <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-f65286b751f121928913d4aa91d94ee9_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#107;\" title=\"Rendered by QuickLaTeX.com\" height=\"12\" width=\"9\" style=\"vertical-align: 0px;\"\/>-dimensional <a href=\"https:\/\/en.wikipedia.org\/wiki\/Linear_subspace\">linear subspace<\/a>).<\/li><\/ul>\n\n\n\n<p>We will only discuss oblivious sketching for this post. We will look at three types of sketching matrices: <a href=\"https:\/\/ar5iv.labs.arxiv.org\/html\/2002.01387#S8.SS3\">Gaussian embeddings<\/a>, <a href=\"https:\/\/ar5iv.labs.arxiv.org\/html\/2002.01387#S9.SS3\">subsampled randomized trignometric transforms<\/a>, and <a href=\"https:\/\/ar5iv.labs.arxiv.org\/html\/1907.09415#Ch1.S2\">sparse sign embeddings<\/a>.<\/p>\n\n\n\n<p>The details of how these sketching matrices are built and their strengths and weaknesses can be a little bit technical. All three constructions are independent from the rest of this article and can be skipped on a first reading. The main point is that good sketching matrices exist and are fast to apply: Reducing <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-054b34698b3706ca9efd09c6723eb8d1_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#98;&#92;&#105;&#110;&#92;&#114;&#101;&#97;&#108;&#94;&#110;\" title=\"Rendered by QuickLaTeX.com\" height=\"13\" width=\"50\" style=\"vertical-align: -1px;\"\/> to <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-efcbcafd69bbeb21d246e35aa3e8bdaf_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#83;&#98;&#92;&#105;&#110;&#92;&#114;&#101;&#97;&#108;&#94;&#123;&#100;&#125;\" title=\"Rendered by QuickLaTeX.com\" height=\"16\" width=\"62\" style=\"vertical-align: -1px;\"\/> requires roughly <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-c08754413e1d82ace2f1ed8aed1140dd_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#92;&#109;&#97;&#116;&#104;&#99;&#97;&#108;&#123;&#79;&#125;&#40;&#110;&#92;&#108;&#111;&#103;&#32;&#110;&#41;\" title=\"Rendered by QuickLaTeX.com\" height=\"19\" width=\"77\" style=\"vertical-align: -5px;\"\/> operations, rather than the <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-9e912c5562eb059da0f46ef6d8bc3b53_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#92;&#109;&#97;&#116;&#104;&#99;&#97;&#108;&#123;&#79;&#125;&#40;&#100;&#110;&#41;\" title=\"Rendered by QuickLaTeX.com\" height=\"19\" width=\"47\" style=\"vertical-align: -5px;\"\/> operations we would expect to multiply a <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-91cf6c6859269c70836c11cac60fc0c2_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#100;&#92;&#116;&#105;&#109;&#101;&#115;&#32;&#110;\" title=\"Rendered by QuickLaTeX.com\" height=\"12\" width=\"42\" style=\"vertical-align: 0px;\"\/> matrix and a vector of length <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-75a5652acadcd645b180a972b75a9d09_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#110;\" title=\"Rendered by QuickLaTeX.com\" height=\"8\" width=\"11\" style=\"vertical-align: 0px;\"\/>.<sup class=\"modern-footnotes-footnote \" data-mfn=\"1\" data-mfn-post-scope=\"000000000000057f0000000000000000_1632\"><a href=\"javascript:void(0)\"  role=\"button\" aria-pressed=\"false\" aria-describedby=\"mfn-content-000000000000057f0000000000000000_1632-1\">1<\/a><\/sup><span id=\"mfn-content-000000000000057f0000000000000000_1632-1\" role=\"tooltip\" class=\"modern-footnotes-footnote__note\" tabindex=\"0\" data-mfn=\"1\">Here, <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-71ee9c94c3d9c97c0db2f438fa86b686_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#92;&#109;&#97;&#116;&#104;&#99;&#97;&#108;&#123;&#79;&#125;&#40;&#92;&#99;&#100;&#111;&#116;&#41;\" title=\"Rendered by QuickLaTeX.com\" height=\"19\" width=\"32\" style=\"vertical-align: -5px;\"\/> is <a href=\"https:\/\/en.wikipedia.org\/wiki\/Big_O_notation\">big O notation<\/a>.<\/span>\n\n\n\n<h3 class=\"wp-block-heading\"><a href=\"https:\/\/ar5iv.labs.arxiv.org\/html\/2002.01387#S8.SS3\">Gaussian Embeddings<\/a><\/h3>\n\n\n\n<p>The simplest type of sketching matrix <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-53c143a41b2d5d2357b8755aff34a28a_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#83;&#92;&#105;&#110;&#92;&#114;&#101;&#97;&#108;&#94;&#123;&#100;&#92;&#116;&#105;&#109;&#101;&#115;&#32;&#110;&#125;\" title=\"Rendered by QuickLaTeX.com\" height=\"16\" width=\"73\" style=\"vertical-align: -1px;\"\/> is obtained by (<a href=\"https:\/\/en.wikipedia.org\/wiki\/Independence_(probability_theory)\">independently<\/a>) setting every entry of <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-b89471a11dd7fd85184a38ddb3ea9145_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#83;\" title=\"Rendered by QuickLaTeX.com\" height=\"12\" width=\"12\" style=\"vertical-align: 0px;\"\/> to be a <a href=\"https:\/\/en.wikipedia.org\/wiki\/Normal_distribution\">Gaussian random number<\/a> with mean zero and variance <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-9636aa8e8b7707e739988afd7e563172_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#49;&#47;&#100;\" title=\"Rendered by QuickLaTeX.com\" height=\"19\" width=\"27\" style=\"vertical-align: -5px;\"\/>. Such a sketching matrix is called a <em>Gaussian embedding<\/em>.<sup class=\"modern-footnotes-footnote \" data-mfn=\"2\" data-mfn-post-scope=\"000000000000057f0000000000000000_1632\"><a href=\"javascript:void(0)\"  role=\"button\" aria-pressed=\"false\" aria-describedby=\"mfn-content-000000000000057f0000000000000000_1632-2\">2<\/a><\/sup><span id=\"mfn-content-000000000000057f0000000000000000_1632-2\" role=\"tooltip\" class=\"modern-footnotes-footnote__note\" tabindex=\"0\" data-mfn=\"2\">Here, <em>embedding<\/em> is a synonym for <em>sketching matrix<\/em>.<\/span>\n\n\n\n<p><strong>Benefits.<\/strong> Gaussian embeddings are simple to code up, requiring only a standard matrix product to apply to a vector <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-bd76151778055d405aee0532cc3a8f23_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#83;&#98;\" title=\"Rendered by QuickLaTeX.com\" height=\"12\" width=\"20\" style=\"vertical-align: 0px;\"\/> or matrix <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-f6b6ffdfa7888fa001ee9e26544c7384_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#83;&#65;\" title=\"Rendered by QuickLaTeX.com\" height=\"13\" width=\"25\" style=\"vertical-align: 0px;\"\/>. Gaussian embeddings admit a <a href=\"https:\/\/ar5iv.labs.arxiv.org\/html\/2002.01387#S8.SS5\">clean theoretical analysis<\/a>, and their mathematical properties are well-understood.<\/p>\n\n\n\n<p><strong>Drawbacks.<\/strong> Computing <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-bd76151778055d405aee0532cc3a8f23_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#83;&#98;\" title=\"Rendered by QuickLaTeX.com\" height=\"12\" width=\"20\" style=\"vertical-align: 0px;\"\/> for a Gaussian embedding costs <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-9e912c5562eb059da0f46ef6d8bc3b53_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#92;&#109;&#97;&#116;&#104;&#99;&#97;&#108;&#123;&#79;&#125;&#40;&#100;&#110;&#41;\" title=\"Rendered by QuickLaTeX.com\" height=\"19\" width=\"47\" style=\"vertical-align: -5px;\"\/> operations, significantly slower than the other sketching matrices we will consider below. Additionally, generating and storing a Gaussian embedding can be computationally expensive.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\"><a href=\"https:\/\/ar5iv.labs.arxiv.org\/html\/2002.01387#S9.SS3\">Subsampled Randomized Trigonometric Transforms<\/a><\/h3>\n\n\n\n<p>The subsampled randomized trigonometric transform (SRTT) sketching matrix takes a more complicated form. The sketching matrix is defined to be a scaled product of three matrices<\/p>\n\n\n\n<p><p class=\"ql-center-displayed-equation\" style=\"line-height: 43px;\"><span class=\"ql-right-eqno\"> &nbsp; <\/span><span class=\"ql-left-eqno\"> &nbsp; <\/span><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-92095cbf9c7d55273bd572a33debe415_l3.png\" height=\"43\" width=\"153\" class=\"ql-img-displayed-equation quicklatex-auto-format\" alt=\"&#92;&#091;&#83;&#32;&#61;&#32;&#92;&#115;&#113;&#114;&#116;&#123;&#92;&#102;&#114;&#97;&#99;&#123;&#110;&#125;&#123;&#100;&#125;&#125;&#32;&#92;&#99;&#100;&#111;&#116;&#32;&#82;&#32;&#92;&#99;&#100;&#111;&#116;&#32;&#70;&#32;&#92;&#99;&#100;&#111;&#116;&#32;&#68;&#46;&#92;&#093;\" title=\"Rendered by QuickLaTeX.com\"\/><\/p><\/p>\n\n\n\n<p>These matrices have the following definitions:<\/p>\n\n\n\n<ul class=\"wp-block-list\"><li><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-a54b8b128e780edcee2fa0d62901e452_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#68;&#92;&#105;&#110;&#92;&#114;&#101;&#97;&#108;&#94;&#123;&#110;&#92;&#116;&#105;&#109;&#101;&#115;&#32;&#110;&#125;\" title=\"Rendered by QuickLaTeX.com\" height=\"14\" width=\"77\" style=\"vertical-align: -1px;\"\/> is a diagonal matrix whose entries are each a random <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-7b7086fe465164ee1c46f7c8b3f43eaf_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#92;&#112;&#109;&#32;&#49;\" title=\"Rendered by QuickLaTeX.com\" height=\"12\" width=\"22\" style=\"vertical-align: 0px;\"\/> (chosen independently with equal probability).<\/li><li><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-670169e1881b0129451718bd02f2d8e7_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#70;&#92;&#105;&#110;&#92;&#114;&#101;&#97;&#108;&#94;&#123;&#110;&#92;&#116;&#105;&#109;&#101;&#115;&#32;&#110;&#125;\" title=\"Rendered by QuickLaTeX.com\" height=\"14\" width=\"76\" style=\"vertical-align: -1px;\"\/> is a fast trigonometric transform such as a fast <a href=\"https:\/\/en.wikipedia.org\/wiki\/Discrete_cosine_transform\">discrete cosine transform<\/a>.<sup class=\"modern-footnotes-footnote \" data-mfn=\"3\" data-mfn-post-scope=\"000000000000057f0000000000000000_1632\"><a href=\"javascript:void(0)\"  role=\"button\" aria-pressed=\"false\" aria-describedby=\"mfn-content-000000000000057f0000000000000000_1632-3\">3<\/a><\/sup><span id=\"mfn-content-000000000000057f0000000000000000_1632-3\" role=\"tooltip\" class=\"modern-footnotes-footnote__note\" tabindex=\"0\" data-mfn=\"3\">One can also use the ordinary <a href=\"https:\/\/www.ethanepperly.com\/index.php\/2021\/05\/10\/big-ideas-in-applied-math-the-fast-fourier-transform\/\">fast Fourier transform<\/a>, but this results in a complex-valued sketch.<\/span><\/li><li><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-bf0e06d53a5f89e22109628a9f6cb379_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#82;&#92;&#105;&#110;&#92;&#114;&#101;&#97;&#108;&#94;&#123;&#100;&#92;&#116;&#105;&#109;&#101;&#115;&#32;&#110;&#125;\" title=\"Rendered by QuickLaTeX.com\" height=\"16\" width=\"74\" style=\"vertical-align: -1px;\"\/> is a <em>selection matrix<\/em>. To generate <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-d08fac616919760e7538df715d3ca0e6_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#82;\" title=\"Rendered by QuickLaTeX.com\" height=\"12\" width=\"14\" style=\"vertical-align: 0px;\"\/>, let <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-252537df37f8ca79038d0721747d56f5_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#105;&#95;&#49;&#44;&#92;&#108;&#100;&#111;&#116;&#115;&#44;&#105;&#95;&#100;\" title=\"Rendered by QuickLaTeX.com\" height=\"16\" width=\"67\" style=\"vertical-align: -4px;\"\/> be a random subset of <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-2b64eb74aa7c11502189aac906c53b96_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#92;&#123;&#49;&#44;&#92;&#108;&#100;&#111;&#116;&#115;&#44;&#110;&#92;&#125;\" title=\"Rendered by QuickLaTeX.com\" height=\"19\" width=\"75\" style=\"vertical-align: -5px;\"\/>, selected without replacement. <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-d08fac616919760e7538df715d3ca0e6_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#82;\" title=\"Rendered by QuickLaTeX.com\" height=\"12\" width=\"14\" style=\"vertical-align: 0px;\"\/> is defined to be a matrix for which <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-1bf39faccf19a656cc8e09a575253feb_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#82;&#98;&#32;&#61;&#32;&#40;&#98;&#95;&#123;&#105;&#95;&#49;&#125;&#44;&#92;&#108;&#100;&#111;&#116;&#115;&#44;&#98;&#95;&#123;&#105;&#95;&#100;&#125;&#41;\" title=\"Rendered by QuickLaTeX.com\" height=\"19\" width=\"137\" style=\"vertical-align: -5px;\"\/> for every vector <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-3ab6f6c64ce17f786a81f8cc8fdcec90_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#98;\" title=\"Rendered by QuickLaTeX.com\" height=\"12\" width=\"8\" style=\"vertical-align: 0px;\"\/>.<\/li><\/ul>\n\n\n\n<p>To store <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-b89471a11dd7fd85184a38ddb3ea9145_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#83;\" title=\"Rendered by QuickLaTeX.com\" height=\"12\" width=\"12\" style=\"vertical-align: 0px;\"\/> on a computer, it is sufficient to store the <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-75a5652acadcd645b180a972b75a9d09_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#110;\" title=\"Rendered by QuickLaTeX.com\" height=\"8\" width=\"11\" style=\"vertical-align: 0px;\"\/> diagonal entries of <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-ab69a4a7716bbf890e5f604a06fd1f13_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#68;\" title=\"Rendered by QuickLaTeX.com\" height=\"12\" width=\"15\" style=\"vertical-align: 0px;\"\/> and the <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-1b031c8458e71f0e4df82ad61d36c0ca_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#100;\" title=\"Rendered by QuickLaTeX.com\" height=\"12\" width=\"10\" style=\"vertical-align: 0px;\"\/> selected coordinates <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-252537df37f8ca79038d0721747d56f5_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#105;&#95;&#49;&#44;&#92;&#108;&#100;&#111;&#116;&#115;&#44;&#105;&#95;&#100;\" title=\"Rendered by QuickLaTeX.com\" height=\"16\" width=\"67\" style=\"vertical-align: -4px;\"\/> defining <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-d08fac616919760e7538df715d3ca0e6_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#82;\" title=\"Rendered by QuickLaTeX.com\" height=\"12\" width=\"14\" style=\"vertical-align: 0px;\"\/>. Multiplication of <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-b89471a11dd7fd85184a38ddb3ea9145_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#83;\" title=\"Rendered by QuickLaTeX.com\" height=\"12\" width=\"12\" style=\"vertical-align: 0px;\"\/> against a vector <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-3ab6f6c64ce17f786a81f8cc8fdcec90_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#98;\" title=\"Rendered by QuickLaTeX.com\" height=\"12\" width=\"8\" style=\"vertical-align: 0px;\"\/> should be carried out by applying each of the matrices <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-d08fac616919760e7538df715d3ca0e6_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#82;\" title=\"Rendered by QuickLaTeX.com\" height=\"12\" width=\"14\" style=\"vertical-align: 0px;\"\/>, <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-7bf5d1207baa8be58658ce9d3cf12043_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#70;\" title=\"Rendered by QuickLaTeX.com\" height=\"12\" width=\"14\" style=\"vertical-align: 0px;\"\/>, and <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-ab69a4a7716bbf890e5f604a06fd1f13_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#68;\" title=\"Rendered by QuickLaTeX.com\" height=\"12\" width=\"15\" style=\"vertical-align: 0px;\"\/> in sequence, such as in the following MATLAB code:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code><mark style=\"background-color:rgba(0, 0, 0, 0)\" class=\"has-inline-color has-vivid-cyan-blue-color\">% Generate randomness for S<\/mark>\nsigns = 2*randi(2,m,1)-3; <mark style=\"background-color:rgba(0, 0, 0, 0)\" class=\"has-inline-color has-vivid-cyan-blue-color\">% diagonal entries of D<\/mark>\nidx = randsample(m,d); <mark style=\"background-color:rgba(0, 0, 0, 0)\" class=\"has-inline-color has-vivid-cyan-blue-color\">% indices i_1,...,i_d defining R<\/mark>\n\n<mark style=\"background-color:rgba(0, 0, 0, 0)\" class=\"has-inline-color has-vivid-cyan-blue-color\">% Multiply S against b<\/mark>\nc = signs .* b <mark style=\"background-color:rgba(0, 0, 0, 0)\" class=\"has-inline-color has-vivid-cyan-blue-color\">% multiply by D<\/mark>\nc = dct(c) <mark style=\"background-color:rgba(0, 0, 0, 0)\" class=\"has-inline-color has-vivid-cyan-blue-color\">% multiply by F<\/mark>\nc = c(idx) <mark style=\"background-color:rgba(0, 0, 0, 0)\" class=\"has-inline-color has-vivid-cyan-blue-color\">% multiply by R<\/mark>\nc = sqrt(n\/d) * c <mark style=\"background-color:rgba(0, 0, 0, 0)\" class=\"has-inline-color has-vivid-cyan-blue-color\">% scale<\/mark><\/code><\/pre>\n\n\n\n<p><strong>Benefits.<\/strong> <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-b89471a11dd7fd85184a38ddb3ea9145_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#83;\" title=\"Rendered by QuickLaTeX.com\" height=\"12\" width=\"12\" style=\"vertical-align: 0px;\"\/> can be applied to a vector <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-3ab6f6c64ce17f786a81f8cc8fdcec90_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#98;\" title=\"Rendered by QuickLaTeX.com\" height=\"12\" width=\"8\" style=\"vertical-align: 0px;\"\/> in <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-e091d48e6c0f5cdaaf8494af69b515b3_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#92;&#109;&#97;&#116;&#104;&#99;&#97;&#108;&#123;&#79;&#125;&#40;&#110;&#32;&#92;&#108;&#111;&#103;&#32;&#110;&#41;\" title=\"Rendered by QuickLaTeX.com\" height=\"19\" width=\"77\" style=\"vertical-align: -5px;\"\/> operations, a significant improvement over the <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-9e912c5562eb059da0f46ef6d8bc3b53_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#92;&#109;&#97;&#116;&#104;&#99;&#97;&#108;&#123;&#79;&#125;&#40;&#100;&#110;&#41;\" title=\"Rendered by QuickLaTeX.com\" height=\"19\" width=\"47\" style=\"vertical-align: -5px;\"\/> cost of a Gaussian embedding. The SRTT has the lowest memory and random number generation requirements of any of the three sketches we discuss in this post.<\/p>\n\n\n\n<p><strong>Drawbacks.<\/strong> Applying <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-b89471a11dd7fd85184a38ddb3ea9145_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#83;\" title=\"Rendered by QuickLaTeX.com\" height=\"12\" width=\"12\" style=\"vertical-align: 0px;\"\/> to a vector requires a good implementation of a fast trigonometric transform. Even with a high-quality trig transform, SRTTs can be significantly slower than sparse sign embeddings (defined below).<sup class=\"modern-footnotes-footnote \" data-mfn=\"4\" data-mfn-post-scope=\"000000000000057f0000000000000000_1632\"><a href=\"javascript:void(0)\"  role=\"button\" aria-pressed=\"false\" aria-describedby=\"mfn-content-000000000000057f0000000000000000_1632-4\">4<\/a><\/sup><span id=\"mfn-content-000000000000057f0000000000000000_1632-4\" role=\"tooltip\" class=\"modern-footnotes-footnote__note\" tabindex=\"0\" data-mfn=\"4\">For an example, see <a href=\"https:\/\/arxiv.org\/abs\/2104.05877\">Figure 2 in this paper<\/a>.<\/span> SRTTs are hard to <a href=\"https:\/\/en.wikipedia.org\/wiki\/Parallel_computing\">parallelize<\/a>.<sup class=\"modern-footnotes-footnote \" data-mfn=\"5\" data-mfn-post-scope=\"000000000000057f0000000000000000_1632\"><a href=\"javascript:void(0)\"  role=\"button\" aria-pressed=\"false\" aria-describedby=\"mfn-content-000000000000057f0000000000000000_1632-5\">5<\/a><\/sup><span id=\"mfn-content-000000000000057f0000000000000000_1632-5\" role=\"tooltip\" class=\"modern-footnotes-footnote__note\" tabindex=\"0\" data-mfn=\"5\"><a href=\"https:\/\/arxiv.org\/abs\/2210.11295\">Block SRTTs<\/a> are more <a href=\"https:\/\/en.wikipedia.org\/wiki\/Parallel_computing\">parallelizable<\/a>, however.<\/span> In theory, the sketching dimension should be chosen to be <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-1488b42ccd4bff1dab4cbe3c9537e8f5_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#100;&#32;&#92;&#97;&#112;&#112;&#114;&#111;&#120;&#32;&#40;&#107;&#92;&#108;&#111;&#103;&#32;&#107;&#41;&#47;&#92;&#118;&#97;&#114;&#101;&#112;&#115;&#105;&#108;&#111;&#110;&#94;&#50;\" title=\"Rendered by QuickLaTeX.com\" height=\"20\" width=\"119\" style=\"vertical-align: -5px;\"\/>, larger than for a Gaussian sketch.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\"><a href=\"https:\/\/ar5iv.labs.arxiv.org\/html\/2002.01387#S9.SS2\">Sparse Sign Embeddings<\/a><\/h3>\n\n\n\n<p>A <em>sparse sign embedding<\/em> takes the form <p class=\"ql-center-displayed-equation\" style=\"line-height: 41px;\"><span class=\"ql-right-eqno\"> &nbsp; <\/span><span class=\"ql-left-eqno\"> &nbsp; <\/span><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-bb5806806e2e5900d4351c08befbd82e_l3.png\" height=\"41\" width=\"209\" class=\"ql-img-displayed-equation quicklatex-auto-format\" alt=\"&#92;&#091;&#83;&#32;&#61;&#32;&#92;&#102;&#114;&#97;&#99;&#123;&#49;&#125;&#123;&#92;&#115;&#113;&#114;&#116;&#123;&#92;&#122;&#101;&#116;&#97;&#125;&#125;&#32;&#92;&#98;&#101;&#103;&#105;&#110;&#123;&#98;&#109;&#97;&#116;&#114;&#105;&#120;&#125;&#32;&#115;&#95;&#49;&#32;&#38;&#32;&#115;&#95;&#50;&#32;&#38;&#32;&#92;&#99;&#100;&#111;&#116;&#115;&#32;&#38;&#32;&#115;&#95;&#110;&#32;&#92;&#101;&#110;&#100;&#123;&#98;&#109;&#97;&#116;&#114;&#105;&#120;&#125;&#46;&#92;&#093;\" title=\"Rendered by QuickLaTeX.com\"\/><\/p>Here, each column <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-2a0fd890dba4d609496d0b299934972e_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#115;&#95;&#105;\" title=\"Rendered by QuickLaTeX.com\" height=\"11\" width=\"13\" style=\"vertical-align: -3px;\"\/> is an independently generated random vector with exactly <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-9eff4ce0824e468ea783c595f12c30ef_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#92;&#122;&#101;&#116;&#97;\" title=\"Rendered by QuickLaTeX.com\" height=\"16\" width=\"9\" style=\"vertical-align: -4px;\"\/> nonzero entries with random <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-7b7086fe465164ee1c46f7c8b3f43eaf_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#92;&#112;&#109;&#32;&#49;\" title=\"Rendered by QuickLaTeX.com\" height=\"12\" width=\"22\" style=\"vertical-align: 0px;\"\/> values in uniformly random positions. The result is a <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-91cf6c6859269c70836c11cac60fc0c2_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#100;&#92;&#116;&#105;&#109;&#101;&#115;&#32;&#110;\" title=\"Rendered by QuickLaTeX.com\" height=\"12\" width=\"42\" style=\"vertical-align: 0px;\"\/> matrix with only <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-99e593205d3c29e66d904a56f366c220_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#92;&#122;&#101;&#116;&#97;&#32;&#92;&#99;&#100;&#111;&#116;&#32;&#110;\" title=\"Rendered by QuickLaTeX.com\" height=\"16\" width=\"33\" style=\"vertical-align: -4px;\"\/> nonzero entries. The parameter <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-9eff4ce0824e468ea783c595f12c30ef_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#92;&#122;&#101;&#116;&#97;\" title=\"Rendered by QuickLaTeX.com\" height=\"16\" width=\"9\" style=\"vertical-align: -4px;\"\/> is often set to a small constant like <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-967b4cb0e15b36ff0727eff922e0a7e7_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#56;\" title=\"Rendered by QuickLaTeX.com\" height=\"12\" width=\"9\" style=\"vertical-align: 0px;\"\/> in practice.<sup class=\"modern-footnotes-footnote \" data-mfn=\"6\" data-mfn-post-scope=\"000000000000057f0000000000000000_1632\"><a href=\"javascript:void(0)\"  role=\"button\" aria-pressed=\"false\" aria-describedby=\"mfn-content-000000000000057f0000000000000000_1632-6\">6<\/a><\/sup><span id=\"mfn-content-000000000000057f0000000000000000_1632-6\" role=\"tooltip\" class=\"modern-footnotes-footnote__note\" tabindex=\"0\" data-mfn=\"6\">This recommendation comes from <a href=\"https:\/\/arxiv.org\/abs\/1902.08651\">the following paper<\/a>, and I&#8217;ve used this parameter setting successfully in my <a href=\"https:\/\/arxiv.org\/abs\/2304.12465\">own<\/a> <a href=\"https:\/\/arxiv.org\/abs\/2311.04362\">work<\/a>.<\/span>\n\n\n\n<p><\/p>\n\n\n\n<p><strong>Benefits.<\/strong> By <a href=\"https:\/\/www.ethanepperly.com\/index.php\/2020\/07\/18\/big-ideas-in-applied-math-sparse-matrices\/\">using a dedicated sparse matrix library<\/a>, <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-b89471a11dd7fd85184a38ddb3ea9145_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#83;\" title=\"Rendered by QuickLaTeX.com\" height=\"12\" width=\"12\" style=\"vertical-align: 0px;\"\/> can be very fast to apply to a vector <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-3ab6f6c64ce17f786a81f8cc8fdcec90_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#98;\" title=\"Rendered by QuickLaTeX.com\" height=\"12\" width=\"8\" style=\"vertical-align: 0px;\"\/> (either <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-546213db83abaec9578680abb1f73f49_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#92;&#109;&#97;&#116;&#104;&#99;&#97;&#108;&#123;&#79;&#125;&#40;&#110;&#41;\" title=\"Rendered by QuickLaTeX.com\" height=\"19\" width=\"38\" style=\"vertical-align: -5px;\"\/> or <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-8230cd8a261745bd109ed9cb9afa5132_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#92;&#109;&#97;&#116;&#104;&#99;&#97;&#108;&#123;&#79;&#125;&#40;&#110;&#92;&#108;&#111;&#103;&#32;&#107;&#41;\" title=\"Rendered by QuickLaTeX.com\" height=\"19\" width=\"76\" style=\"vertical-align: -5px;\"\/> operations) to apply to a vector, depending on parameter choices (see below). With a good sparse matrix library, sparse sign embeddings are often the fastest sketching matrix by a wide margin.<\/p>\n\n\n\n<p><strong>Drawbacks.<\/strong> To be fast, sparse sign embeddings requires a good sparse matrix library. They require generating and storing roughly <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-487c856c45634151bc1a1fac621f0287_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#92;&#122;&#101;&#116;&#97;&#32;&#110;\" title=\"Rendered by QuickLaTeX.com\" height=\"16\" width=\"20\" style=\"vertical-align: -4px;\"\/> random numbers, higher than SRTTs (roughly <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-75a5652acadcd645b180a972b75a9d09_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#110;\" title=\"Rendered by QuickLaTeX.com\" height=\"8\" width=\"11\" style=\"vertical-align: 0px;\"\/> numbers) but much less than Gaussian embeddings (<img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-79b55409ea8d5fad235e919154b612ab_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#100;&#110;\" title=\"Rendered by QuickLaTeX.com\" height=\"12\" width=\"20\" style=\"vertical-align: 0px;\"\/> numbers). <a href=\"https:\/\/epubs.siam.org\/doi\/abs\/10.1137\/1.9781611974331.ch21\">In theory<\/a>, the sketching dimension should be chosen to be <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-1488b42ccd4bff1dab4cbe3c9537e8f5_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#100;&#32;&#92;&#97;&#112;&#112;&#114;&#111;&#120;&#32;&#40;&#107;&#92;&#108;&#111;&#103;&#32;&#107;&#41;&#47;&#92;&#118;&#97;&#114;&#101;&#112;&#115;&#105;&#108;&#111;&#110;&#94;&#50;\" title=\"Rendered by QuickLaTeX.com\" height=\"20\" width=\"119\" style=\"vertical-align: -5px;\"\/> and the sparsity should be set to <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-7a487f5794c6d73e51d0ce0d7cad368f_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#92;&#122;&#101;&#116;&#97;&#32;&#92;&#97;&#112;&#112;&#114;&#111;&#120;&#32;&#40;&#92;&#108;&#111;&#103;&#32;&#107;&#41;&#47;&#92;&#118;&#97;&#114;&#101;&#112;&#115;&#105;&#108;&#111;&#110;\" title=\"Rendered by QuickLaTeX.com\" height=\"19\" width=\"99\" style=\"vertical-align: -5px;\"\/>; the theoretically sanctioned sketching dimension (at least according to existing theory) is larger than for a Gaussian sketch. In practice, we can often get away with using <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-6774052ff5d917bba64006b63b9e31e0_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#100;&#32;&#92;&#97;&#112;&#112;&#114;&#111;&#120;&#32;&#107;&#47;&#92;&#118;&#97;&#114;&#101;&#112;&#115;&#105;&#108;&#111;&#110;&#94;&#50;\" title=\"Rendered by QuickLaTeX.com\" height=\"20\" width=\"67\" style=\"vertical-align: -5px;\"\/> and <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-cbaf4db2dd440a7cbec3fcacc40f0ee8_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#92;&#122;&#101;&#116;&#97;&#61;&#56;\" title=\"Rendered by QuickLaTeX.com\" height=\"16\" width=\"42\" style=\"vertical-align: -4px;\"\/>.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Summary<\/h3>\n\n\n\n<p>Using either SRTTs or sparse maps, a sketching a vector of length <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-75a5652acadcd645b180a972b75a9d09_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#110;\" title=\"Rendered by QuickLaTeX.com\" height=\"8\" width=\"11\" style=\"vertical-align: 0px;\"\/> down to <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-1b031c8458e71f0e4df82ad61d36c0ca_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#100;\" title=\"Rendered by QuickLaTeX.com\" height=\"12\" width=\"10\" style=\"vertical-align: 0px;\"\/> dimensions requires only <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-546213db83abaec9578680abb1f73f49_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#92;&#109;&#97;&#116;&#104;&#99;&#97;&#108;&#123;&#79;&#125;&#40;&#110;&#41;\" title=\"Rendered by QuickLaTeX.com\" height=\"19\" width=\"38\" style=\"vertical-align: -5px;\"\/> to <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-c08754413e1d82ace2f1ed8aed1140dd_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#92;&#109;&#97;&#116;&#104;&#99;&#97;&#108;&#123;&#79;&#125;&#40;&#110;&#92;&#108;&#111;&#103;&#32;&#110;&#41;\" title=\"Rendered by QuickLaTeX.com\" height=\"19\" width=\"77\" style=\"vertical-align: -5px;\"\/> operations. To apply a sketch to an entire <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-1a55278ecc061a49dac84a0a29899010_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#110;&#92;&#116;&#105;&#109;&#101;&#115;&#32;&#107;\" title=\"Rendered by QuickLaTeX.com\" height=\"12\" width=\"41\" style=\"vertical-align: 0px;\"\/> matrix <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-11f4f587954b361e7d78940f65b8d70d_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#65;\" title=\"Rendered by QuickLaTeX.com\" height=\"13\" width=\"13\" style=\"vertical-align: 0px;\"\/> thus requires roughly <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-41d46af2cd66ec65c957f8de268ae1a4_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#92;&#109;&#97;&#116;&#104;&#99;&#97;&#108;&#123;&#79;&#125;&#40;&#110;&#107;&#41;\" title=\"Rendered by QuickLaTeX.com\" height=\"19\" width=\"47\" style=\"vertical-align: -5px;\"\/> operations. Therefore, sketching offers the promise of speeding up linear algebraic computations involving <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-11f4f587954b361e7d78940f65b8d70d_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#65;\" title=\"Rendered by QuickLaTeX.com\" height=\"13\" width=\"13\" style=\"vertical-align: 0px;\"\/>, which <a href=\"https:\/\/en.wikipedia.org\/wiki\/QR_decomposition\">typically<\/a> <a href=\"https:\/\/en.wikipedia.org\/wiki\/Singular_value_decomposition\">take<\/a> <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-ebbc5e7bbff1376cf1e3f7b3220232a9_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#92;&#109;&#97;&#116;&#104;&#99;&#97;&#108;&#123;&#79;&#125;&#40;&#110;&#107;&#94;&#50;&#41;\" title=\"Rendered by QuickLaTeX.com\" height=\"20\" width=\"55\" style=\"vertical-align: -5px;\"\/> operations.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">How Can You Use Sketching?<\/h2>\n\n\n\n<p>The simplest way to use sketching is to first apply the sketch to dimensionality-reduce all of your data and then apply a standard algorithm to solve the problem using the reduced data. This approach to using sketching is called <em><a href=\"https:\/\/ar5iv.labs.arxiv.org\/html\/2002.01387#S10.SS3\">sketch-and-solve<\/a><\/em>.<\/p>\n\n\n\n<p>As an example, let\u2019s apply sketch-and-solve to the <em><a href=\"https:\/\/en.m.wikipedia.org\/wiki\/Least_squares\">least-squares problem<\/a><\/em>:<p class=\"ql-center-displayed-equation\" style=\"line-height: 28px;\"><span class=\"ql-right-eqno\"> (2) <\/span><span class=\"ql-left-eqno\"> &nbsp; <\/span><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-07528304ce875f54aa1173062c5ec10a_l3.png\" height=\"28\" width=\"151\" class=\"ql-img-displayed-equation quicklatex-auto-format\" alt=\"&#92;&#091;&#92;&#111;&#112;&#101;&#114;&#97;&#116;&#111;&#114;&#110;&#97;&#109;&#101;&#42;&#123;&#109;&#105;&#110;&#105;&#109;&#105;&#122;&#101;&#125;&#95;&#123;&#120;&#92;&#105;&#110;&#92;&#114;&#101;&#97;&#108;&#94;&#107;&#125;&#32;&#92;&#110;&#111;&#114;&#109;&#123;&#65;&#120;&#32;&#45;&#32;&#98;&#125;&#46;&#32;&#92;&#093;\" title=\"Rendered by QuickLaTeX.com\"\/><\/p>We assume this problem is highly overdetermined with <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-11f4f587954b361e7d78940f65b8d70d_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#65;\" title=\"Rendered by QuickLaTeX.com\" height=\"13\" width=\"13\" style=\"vertical-align: 0px;\"\/> having many more rows <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-75a5652acadcd645b180a972b75a9d09_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#110;\" title=\"Rendered by QuickLaTeX.com\" height=\"8\" width=\"11\" style=\"vertical-align: 0px;\"\/> than columns <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-6d890b0f5af56bc2bde465a9af2fd218_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#109;\" title=\"Rendered by QuickLaTeX.com\" height=\"8\" width=\"15\" style=\"vertical-align: 0px;\"\/>.<\/p>\n\n\n\n<p>To solve this problem with sketch-and-solve, generate a good sketching matrix <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-b89471a11dd7fd85184a38ddb3ea9145_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#83;\" title=\"Rendered by QuickLaTeX.com\" height=\"12\" width=\"12\" style=\"vertical-align: 0px;\"\/> for the set <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-38b6221cea998a6523a5ff5ac6835eb3_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#92;&#109;&#97;&#116;&#104;&#115;&#102;&#123;&#69;&#125;&#32;&#61;&#32;&#92;&#111;&#112;&#101;&#114;&#97;&#116;&#111;&#114;&#110;&#97;&#109;&#101;&#123;&#99;&#111;&#108;&#125;&#40;&#92;&#111;&#110;&#101;&#98;&#121;&#116;&#119;&#111;&#123;&#65;&#125;&#123;&#98;&#125;&#41;\" title=\"Rendered by QuickLaTeX.com\" height=\"22\" width=\"120\" style=\"vertical-align: -7px;\"\/>. Applying <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-b89471a11dd7fd85184a38ddb3ea9145_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#83;\" title=\"Rendered by QuickLaTeX.com\" height=\"12\" width=\"12\" style=\"vertical-align: 0px;\"\/> to our data <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-11f4f587954b361e7d78940f65b8d70d_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#65;\" title=\"Rendered by QuickLaTeX.com\" height=\"13\" width=\"13\" style=\"vertical-align: 0px;\"\/> and <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-3ab6f6c64ce17f786a81f8cc8fdcec90_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#98;\" title=\"Rendered by QuickLaTeX.com\" height=\"12\" width=\"8\" style=\"vertical-align: 0px;\"\/>, we get a dimensionality-reduced least-squares problem <p class=\"ql-center-displayed-equation\" style=\"line-height: 28px;\"><span class=\"ql-right-eqno\"> (3) <\/span><span class=\"ql-left-eqno\"> &nbsp; <\/span><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-7110d3e9c1dd7e29793603a5bafdd201_l3.png\" height=\"28\" width=\"188\" class=\"ql-img-displayed-equation quicklatex-auto-format\" alt=\"&#92;&#091;&#92;&#111;&#112;&#101;&#114;&#97;&#116;&#111;&#114;&#110;&#97;&#109;&#101;&#42;&#123;&#109;&#105;&#110;&#105;&#109;&#105;&#122;&#101;&#125;&#95;&#123;&#92;&#104;&#97;&#116;&#123;&#120;&#125;&#92;&#105;&#110;&#92;&#114;&#101;&#97;&#108;&#94;&#107;&#125;&#32;&#92;&#110;&#111;&#114;&#109;&#123;&#40;&#83;&#65;&#41;&#92;&#104;&#97;&#116;&#123;&#120;&#125;&#32;&#45;&#32;&#83;&#98;&#125;&#46;&#32;&#92;&#093;\" title=\"Rendered by QuickLaTeX.com\"\/><\/p>The solution <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-aaab1105f322d56886350f3bdc22c2b8_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#92;&#104;&#97;&#116;&#123;&#120;&#125;\" title=\"Rendered by QuickLaTeX.com\" height=\"14\" width=\"11\" style=\"vertical-align: 0px;\"\/> is the sketch-and-solve solution to the least-squares problem, which we can use as an approximate solution to the original least-squares problem.<\/p>\n\n\n\n<p>Least-squares is just one example of the sketch-and-solve paradigm. We can also use sketching to accelerate other algorithms. For instance, we could apply sketch-and-solve to <a href=\"https:\/\/en.m.wikipedia.org\/wiki\/Cluster_analysis\">clustering<\/a>. To cluster data points <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-c319de52694e3ec71c4da7143e575998_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#120;&#95;&#49;&#44;&#92;&#108;&#100;&#111;&#116;&#115;&#44;&#120;&#95;&#112;\" title=\"Rendered by QuickLaTeX.com\" height=\"14\" width=\"74\" style=\"vertical-align: -6px;\"\/>, first apply sketching to obtain <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-cdbd3bcfbe479fcefeb48510ffc16c30_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#83;&#120;&#95;&#49;&#44;&#92;&#108;&#100;&#111;&#116;&#115;&#44;&#83;&#120;&#95;&#112;\" title=\"Rendered by QuickLaTeX.com\" height=\"18\" width=\"98\" style=\"vertical-align: -6px;\"\/> and then apply an out-of-the-box clustering algorithms like <a href=\"https:\/\/en.m.wikipedia.org\/wiki\/K-means_clustering#:~:text=k%2Dmeans%20clustering%20is%20a,a%20prototype%20of%20the%20cluster.\">k-means<\/a> to the sketched data points.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Does Sketching Work?<\/h2>\n\n\n\n<p>Most often, when sketching critics say &#8220;sketching doesn&#8217;t work&#8221;, what they mean is &#8220;sketch-and-solve doesn&#8217;t work&#8221;.<\/p>\n\n\n\n<p>To address this question in a more concrete setting, let&#8217;s go back to the least-squares problem (2). Let <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-4c269341681e03c841aa0aee1e9af79d_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#120;&#95;&#92;&#115;&#116;&#97;&#114;\" title=\"Rendered by QuickLaTeX.com\" height=\"11\" width=\"17\" style=\"vertical-align: -3px;\"\/> denote the optimal least-squares solution and let <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-aaab1105f322d56886350f3bdc22c2b8_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#92;&#104;&#97;&#116;&#123;&#120;&#125;\" title=\"Rendered by QuickLaTeX.com\" height=\"14\" width=\"11\" style=\"vertical-align: 0px;\"\/> be the sketch-and-solve solution (3). Then, using the distortion condition (1), one can show that <p class=\"ql-center-displayed-equation\" style=\"line-height: 36px;\"><span class=\"ql-right-eqno\"> &nbsp; <\/span><span class=\"ql-left-eqno\"> &nbsp; <\/span><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-3d1d034bd8bf93f1c15e435a27a355ee_l3.png\" height=\"36\" width=\"215\" class=\"ql-img-displayed-equation quicklatex-auto-format\" alt=\"&#92;&#091;&#92;&#110;&#111;&#114;&#109;&#123;&#65;&#92;&#104;&#97;&#116;&#123;&#120;&#125;&#32;&#45;&#32;&#98;&#125;&#32;&#92;&#108;&#101;&#32;&#92;&#102;&#114;&#97;&#99;&#123;&#49;&#43;&#92;&#118;&#97;&#114;&#101;&#112;&#115;&#105;&#108;&#111;&#110;&#125;&#123;&#49;&#45;&#92;&#118;&#97;&#114;&#101;&#112;&#115;&#105;&#108;&#111;&#110;&#125;&#32;&#92;&#110;&#111;&#114;&#109;&#123;&#65;&#120;&#32;&#45;&#32;&#98;&#125;&#46;&#92;&#093;\" title=\"Rendered by QuickLaTeX.com\"\/><\/p>If we use a sketching matrix with a distortion of <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-1d13d4417e8bc0b66779f0c2aa1e2c6f_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#92;&#118;&#97;&#114;&#101;&#112;&#115;&#105;&#108;&#111;&#110;&#32;&#61;&#32;&#49;&#47;&#51;\" title=\"Rendered by QuickLaTeX.com\" height=\"19\" width=\"59\" style=\"vertical-align: -5px;\"\/>, then this bound tells us that <p class=\"ql-center-displayed-equation\" style=\"line-height: 19px;\"><span class=\"ql-right-eqno\"> (4) <\/span><span class=\"ql-left-eqno\"> &nbsp; <\/span><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-588170743415f491f48ef2adc3d46628_l3.png\" height=\"19\" width=\"189\" class=\"ql-img-displayed-equation quicklatex-auto-format\" alt=\"&#92;&#091;&#92;&#110;&#111;&#114;&#109;&#123;&#65;&#92;&#104;&#97;&#116;&#123;&#120;&#125;&#32;&#45;&#32;&#98;&#125;&#32;&#92;&#108;&#101;&#32;&#50;&#92;&#110;&#111;&#114;&#109;&#123;&#65;&#120;&#95;&#92;&#115;&#116;&#97;&#114;&#32;&#45;&#32;&#98;&#125;&#46;&#32;&#92;&#093;\" title=\"Rendered by QuickLaTeX.com\"\/><\/p><\/p>\n\n\n\n<p>Is this a good result or a bad result? Ultimately, it depends. In some applications, the quality of a putative least-squares solution <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-aaab1105f322d56886350f3bdc22c2b8_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#92;&#104;&#97;&#116;&#123;&#120;&#125;\" title=\"Rendered by QuickLaTeX.com\" height=\"14\" width=\"11\" style=\"vertical-align: 0px;\"\/> is can be assessed from the residual norm <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-d3ead3f4323e66056696f0f89dc30488_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#92;&#110;&#111;&#114;&#109;&#123;&#65;&#92;&#104;&#97;&#116;&#123;&#120;&#125;&#32;&#45;&#32;&#98;&#125;\" title=\"Rendered by QuickLaTeX.com\" height=\"19\" width=\"67\" style=\"vertical-align: -5px;\"\/>. For such applications, the bound (4) ensures that <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-d3ead3f4323e66056696f0f89dc30488_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#92;&#110;&#111;&#114;&#109;&#123;&#65;&#92;&#104;&#97;&#116;&#123;&#120;&#125;&#32;&#45;&#32;&#98;&#125;\" title=\"Rendered by QuickLaTeX.com\" height=\"19\" width=\"67\" style=\"vertical-align: -5px;\"\/> is at most twice <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-2b3cf5f9a0edf5efc3947606f2f7804d_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#92;&#110;&#111;&#114;&#109;&#123;&#65;&#120;&#95;&#92;&#115;&#116;&#97;&#114;&#45;&#98;&#125;\" title=\"Rendered by QuickLaTeX.com\" height=\"19\" width=\"75\" style=\"vertical-align: -5px;\"\/>. Often, this means <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-aaab1105f322d56886350f3bdc22c2b8_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#92;&#104;&#97;&#116;&#123;&#120;&#125;\" title=\"Rendered by QuickLaTeX.com\" height=\"14\" width=\"11\" style=\"vertical-align: 0px;\"\/> is a pretty decent approximate solution to the least-squares problem.<\/p>\n\n\n\n<p>For other problems, the appropriate measure of accuracy is the so-called <em>forward error<\/em> <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-98c8b48d53691aa22deb9307b7a93265_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#92;&#110;&#111;&#114;&#109;&#123;&#92;&#104;&#97;&#116;&#123;&#120;&#125;&#32;&#45;&#32;&#120;&#95;&#92;&#115;&#116;&#97;&#114;&#125;\" title=\"Rendered by QuickLaTeX.com\" height=\"19\" width=\"63\" style=\"vertical-align: -5px;\"\/>, measuring how close <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-aaab1105f322d56886350f3bdc22c2b8_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#92;&#104;&#97;&#116;&#123;&#120;&#125;\" title=\"Rendered by QuickLaTeX.com\" height=\"14\" width=\"11\" style=\"vertical-align: 0px;\"\/> is to <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-4c269341681e03c841aa0aee1e9af79d_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#120;&#95;&#92;&#115;&#116;&#97;&#114;\" title=\"Rendered by QuickLaTeX.com\" height=\"11\" width=\"17\" style=\"vertical-align: -3px;\"\/>. For these cases, it is possible that <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-98c8b48d53691aa22deb9307b7a93265_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#92;&#110;&#111;&#114;&#109;&#123;&#92;&#104;&#97;&#116;&#123;&#120;&#125;&#32;&#45;&#32;&#120;&#95;&#92;&#115;&#116;&#97;&#114;&#125;\" title=\"Rendered by QuickLaTeX.com\" height=\"19\" width=\"63\" style=\"vertical-align: -5px;\"\/> might be large even though the residuals are comparable (4).<\/p>\n\n\n\n<p>Let&#8217;s see an example, <a href=\"https:\/\/github.com\/eepperly\/Iterative-Sketching-Is-Stable\">using the MATLAB code from my paper<\/a>:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>&#091;A, b, x, r] = random_ls_problem(1e4, 1e2, 1e8, 1e-4); <mark style=\"background-color:rgba(0, 0, 0, 0)\" class=\"has-inline-color has-vivid-cyan-blue-color\">% Random LS problem<\/mark>\nS = sparsesign(4e2, 1e4, 8); <mark style=\"background-color:rgba(0, 0, 0, 0)\" class=\"has-inline-color has-vivid-cyan-blue-color\">% Sparse sign embedding<\/mark>\nsketch_and_solve = (S*A) \\ (S*b); <mark style=\"background-color:rgba(0, 0, 0, 0)\" class=\"has-inline-color has-vivid-cyan-blue-color\">% Sketch-and-solve<\/mark>\ndirect = A \\ b; <mark style=\"background-color:rgba(0, 0, 0, 0)\" class=\"has-inline-color has-vivid-cyan-blue-color\">% MATLAB mldivide<\/mark><\/code><\/pre>\n\n\n\n<p>Here, we generate a random least-squares problem of size 10,000 by 100 (with condition number <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-ca98a518accbf93c39e9b520185d3bb3_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#49;&#48;&#94;&#56;\" title=\"Rendered by QuickLaTeX.com\" height=\"15\" width=\"24\" style=\"vertical-align: 0px;\"\/> and residual norm <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-049bee80c6164b97a3d9687b8abc045b_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#49;&#48;&#94;&#123;&#45;&#52;&#125;\" title=\"Rendered by QuickLaTeX.com\" height=\"15\" width=\"35\" style=\"vertical-align: 0px;\"\/>). Then, we generate a sparse sign embedding of dimension <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-8d1006c4b3e5854d6d60b711ebc6c779_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#100;&#32;&#61;&#32;&#52;&#48;&#48;\" title=\"Rendered by QuickLaTeX.com\" height=\"12\" width=\"60\" style=\"vertical-align: 0px;\"\/> (corresponding to a distortion of roughly <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-729b171ad5f3446bb622083e3d78bccb_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#92;&#118;&#97;&#114;&#101;&#112;&#115;&#105;&#108;&#111;&#110;&#32;&#92;&#97;&#112;&#112;&#114;&#111;&#120;&#32;&#49;&#47;&#50;\" title=\"Rendered by QuickLaTeX.com\" height=\"19\" width=\"58\" style=\"vertical-align: -5px;\"\/>). Then, we compute the sketch-and-solve solution and, as reference, a &#8220;direct&#8221; solution by <a href=\"https:\/\/www.mathworks.com\/help\/matlab\/ref\/mldivide.html\">MATLAB&#8217;s \\<\/a>.<\/p>\n\n\n\n<p>We compare the quality of the sketch-and-solve solution to the direct solution, using both the residual and forward error:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>fprintf('Residuals: sketch-and-solve %.2e, direct %.2e, optimal %.2e\\n',...\n           norm(b-A*sketch_and_solve), norm(b-A*direct), norm(r))\nfprintf('Forward errors: sketch-and-solve %.2e, direct %.2e\\n',...\n           norm(x-sketch_and_solve), norm(x-direct))<\/code><\/pre>\n\n\n\n<p>Here&#8217;s the output:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>Residuals: sketch-and-solve 1.13e-04, direct 1.00e-04, optimal 1.00e-04\nForward errors: sketch-and-solve 1.06e+03, direct 8.08e-07<\/code><\/pre>\n\n\n\n<p>The sketch-and-solve solution has a residual norm of <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-a813a12f073ce5751457ed4db750f7b7_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#49;&#46;&#49;&#51;&#92;&#116;&#105;&#109;&#101;&#115;&#32;&#49;&#48;&#94;&#123;&#45;&#52;&#125;\" title=\"Rendered by QuickLaTeX.com\" height=\"15\" width=\"88\" style=\"vertical-align: 0px;\"\/>, close to direct method&#8217;s residual norm of <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-a652b93d851819c8195b76d98071f9f2_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#49;&#46;&#48;&#48;&#92;&#116;&#105;&#109;&#101;&#115;&#32;&#49;&#48;&#94;&#123;&#45;&#52;&#125;\" title=\"Rendered by QuickLaTeX.com\" height=\"15\" width=\"88\" style=\"vertical-align: 0px;\"\/>. However, the forward error of sketch-and-solve is <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-413f193dd84a48c5e1c50e498cf1c936_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#49;&#92;&#116;&#105;&#109;&#101;&#115;&#32;&#49;&#48;&#94;&#51;\" title=\"Rendered by QuickLaTeX.com\" height=\"15\" width=\"55\" style=\"vertical-align: 0px;\"\/> <em>nine orders of magnitude larger<\/em> than the direct method&#8217;s forward error of <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-060458e9b9ad2279dc0ee06a2482e69a_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#56;&#92;&#116;&#105;&#109;&#101;&#115;&#32;&#49;&#48;&#94;&#123;&#45;&#55;&#125;\" title=\"Rendered by QuickLaTeX.com\" height=\"15\" width=\"67\" style=\"vertical-align: 0px;\"\/>.<\/p>\n\n\n\n<p>Does sketch-and-solve work? Ultimately, it&#8217;s a question of what kind of accuracy you need for your application. If a small-enough residual is all that&#8217;s needed, then sketch-and-solve is perfectly adequate. If small forward error is needed, sketch-and-solve can be quite bad.<\/p>\n\n\n\n<p>One way sketch-and-solve can be improved is by increasing the sketching dimension <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-1b031c8458e71f0e4df82ad61d36c0ca_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#100;\" title=\"Rendered by QuickLaTeX.com\" height=\"12\" width=\"10\" style=\"vertical-align: 0px;\"\/> and lowering the distortion <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-68eb29837031f4afbd699052143a2826_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#92;&#118;&#97;&#114;&#101;&#112;&#115;&#105;&#108;&#111;&#110;\" title=\"Rendered by QuickLaTeX.com\" height=\"8\" width=\"8\" style=\"vertical-align: 0px;\"\/>. Unfortunately, improving the distortion of the sketch is expensive. Because of the relation <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-91bbfa6c09e0ddb8337623b38376321a_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#100;&#32;&#92;&#97;&#112;&#112;&#114;&#111;&#120;&#32;&#107;&#32;&#47;&#92;&#118;&#97;&#114;&#101;&#112;&#115;&#105;&#108;&#111;&#110;&#94;&#50;\" title=\"Rendered by QuickLaTeX.com\" height=\"20\" width=\"67\" style=\"vertical-align: -5px;\"\/>, to decrease the distortion by a factor of ten requires increasing the sketching dimension <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-1b031c8458e71f0e4df82ad61d36c0ca_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#100;\" title=\"Rendered by QuickLaTeX.com\" height=\"12\" width=\"10\" style=\"vertical-align: 0px;\"\/> by a factor of one hundred! Thus, sketch-and-solve is really only appropriate when a low degree of distortion <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-68eb29837031f4afbd699052143a2826_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#92;&#118;&#97;&#114;&#101;&#112;&#115;&#105;&#108;&#111;&#110;\" title=\"Rendered by QuickLaTeX.com\" height=\"8\" width=\"8\" style=\"vertical-align: 0px;\"\/> is necessary.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Iterative Sketching: Combining Sketching with Iteration<\/h2>\n\n\n\n<p>Sketch-and-solve is a fast way to get a low-accuracy solution to a least-squares problem. But it&#8217;s not the only way to use sketching for least-squares. One can also use sketching to obtain high-accuracy solutions by combining sketching with an <em><a href=\"https:\/\/en.m.wikipedia.org\/wiki\/Iterative_method\">iterative method<\/a><\/em>.<\/p>\n\n\n\n<p>There are many iterative methods for least-square problems. Iterative methods generate a sequence of approximate solutions <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-5438a09a21bf0ae25500f639566aa505_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#120;&#95;&#49;&#44;&#120;&#95;&#50;&#44;&#92;&#108;&#100;&#111;&#116;&#115;\" title=\"Rendered by QuickLaTeX.com\" height=\"12\" width=\"71\" style=\"vertical-align: -4px;\"\/> that we hope will converge at a rapid rate to the true least-squares solution, <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-4c269341681e03c841aa0aee1e9af79d_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#120;&#95;&#92;&#115;&#116;&#97;&#114;\" title=\"Rendered by QuickLaTeX.com\" height=\"11\" width=\"17\" style=\"vertical-align: -3px;\"\/>.<\/p>\n\n\n\n<p>To using sketching to solve least-squares problems iteratively, we can use the following observation:<\/p>\n\n\n\n<blockquote class=\"wp-block-quote is-layout-flow wp-block-quote-is-layout-flow\"><p>If <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-b89471a11dd7fd85184a38ddb3ea9145_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#83;\" title=\"Rendered by QuickLaTeX.com\" height=\"12\" width=\"12\" style=\"vertical-align: 0px;\"\/> is a sketching matrix for <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-f1adbeeb31f7ea185b8368f38f98f186_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#92;&#109;&#97;&#116;&#104;&#115;&#102;&#123;&#69;&#125;&#32;&#61;&#32;&#92;&#111;&#112;&#101;&#114;&#97;&#116;&#111;&#114;&#110;&#97;&#109;&#101;&#123;&#99;&#111;&#108;&#125;&#40;&#65;&#41;\" title=\"Rendered by QuickLaTeX.com\" height=\"19\" width=\"81\" style=\"vertical-align: -5px;\"\/>, then <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-18eb450bcd37618953fb51115f4bd6ad_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#40;&#83;&#65;&#41;&#94;&#92;&#116;&#111;&#112;&#32;&#83;&#65;&#32;&#92;&#97;&#112;&#112;&#114;&#111;&#120;&#32;&#65;&#94;&#92;&#116;&#111;&#112;&#32;&#65;\" title=\"Rendered by QuickLaTeX.com\" height=\"20\" width=\"136\" style=\"vertical-align: -5px;\"\/>.<\/p><\/blockquote>\n\n\n\n<p>Therefore, if we compute a QR factorization <p class=\"ql-center-displayed-equation\" style=\"line-height: 17px;\"><span class=\"ql-right-eqno\"> &nbsp; <\/span><span class=\"ql-left-eqno\"> &nbsp; <\/span><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-9edf84077821e9e0cd0c772b77024799_l3.png\" height=\"17\" width=\"81\" class=\"ql-img-displayed-equation quicklatex-auto-format\" alt=\"&#92;&#091;&#83;&#65;&#32;&#61;&#32;&#81;&#82;&#44;&#92;&#093;\" title=\"Rendered by QuickLaTeX.com\"\/><\/p>then <p class=\"ql-center-displayed-equation\" style=\"line-height: 22px;\"><span class=\"ql-right-eqno\"> &nbsp; <\/span><span class=\"ql-left-eqno\"> &nbsp; <\/span><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-90e14ee1172525fd795980cb3ef1a647_l3.png\" height=\"22\" width=\"319\" class=\"ql-img-displayed-equation quicklatex-auto-format\" alt=\"&#92;&#091;&#65;&#94;&#92;&#116;&#111;&#112;&#32;&#65;&#32;&#92;&#97;&#112;&#112;&#114;&#111;&#120;&#32;&#40;&#83;&#65;&#41;&#94;&#92;&#116;&#111;&#112;&#32;&#40;&#83;&#65;&#41;&#32;&#61;&#32;&#82;&#94;&#92;&#116;&#111;&#112;&#32;&#81;&#94;&#92;&#116;&#111;&#112;&#32;&#81;&#32;&#82;&#32;&#61;&#32;&#82;&#94;&#92;&#116;&#111;&#112;&#32;&#82;&#46;&#92;&#093;\" title=\"Rendered by QuickLaTeX.com\"\/><\/p>Notice that we used the fact that <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-c2027d18e5c4de036513eb465647de0d_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#81;&#94;&#92;&#116;&#111;&#112;&#32;&#81;&#32;&#61;&#32;&#73;\" title=\"Rendered by QuickLaTeX.com\" height=\"19\" width=\"72\" style=\"vertical-align: -4px;\"\/> since <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-e7f252699e1616398a7ce5179d3e425e_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#81;\" title=\"Rendered by QuickLaTeX.com\" height=\"16\" width=\"14\" style=\"vertical-align: -4px;\"\/> has orthonormal columns. The conclusion is that <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-30920042bab17632f6d5c08ae35d150e_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#82;&#94;&#92;&#116;&#111;&#112;&#32;&#82;&#32;&#92;&#97;&#112;&#112;&#114;&#111;&#120;&#32;&#65;&#94;&#92;&#116;&#111;&#112;&#32;&#65;\" title=\"Rendered by QuickLaTeX.com\" height=\"15\" width=\"100\" style=\"vertical-align: 0px;\"\/>.<\/p>\n\n\n\n<p>Let\u2019s use the approximation <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-30920042bab17632f6d5c08ae35d150e_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#82;&#94;&#92;&#116;&#111;&#112;&#32;&#82;&#32;&#92;&#97;&#112;&#112;&#114;&#111;&#120;&#32;&#65;&#94;&#92;&#116;&#111;&#112;&#32;&#65;\" title=\"Rendered by QuickLaTeX.com\" height=\"15\" width=\"100\" style=\"vertical-align: 0px;\"\/> to solve the least-squares problem iteratively. Start off with the normal equations<sup class=\"modern-footnotes-footnote \" data-mfn=\"7\" data-mfn-post-scope=\"000000000000057f0000000000000000_1632\"><a href=\"javascript:void(0)\"  role=\"button\" aria-pressed=\"false\" aria-describedby=\"mfn-content-000000000000057f0000000000000000_1632-7\">7<\/a><\/sup><span id=\"mfn-content-000000000000057f0000000000000000_1632-7\" role=\"tooltip\" class=\"modern-footnotes-footnote__note\" tabindex=\"0\" data-mfn=\"7\">As I&#8217;ve described in a <a href=\"https:\/\/www.ethanepperly.com\/index.php\/2022\/07\/26\/dont-solve-the-normal-equations\/\">previous post<\/a>, it&#8217;s generally inadvisable to solve least-squares problems using the normal equations. Here, we&#8217;re just using the normal equations as a conceptual tool to derive an algorithm for solving the least-squares problem.<\/span><p class=\"ql-center-displayed-equation\" style=\"line-height: 22px;\"><span class=\"ql-right-eqno\"> (5) <\/span><span class=\"ql-left-eqno\"> &nbsp; <\/span><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-3f3c5265345a169d6633a759854510da_l3.png\" height=\"22\" width=\"122\" class=\"ql-img-displayed-equation quicklatex-auto-format\" alt=\"&#92;&#091;&#40;&#65;&#94;&#92;&#116;&#111;&#112;&#32;&#65;&#41;&#120;&#32;&#61;&#32;&#65;&#94;&#92;&#116;&#111;&#112;&#32;&#98;&#46;&#32;&#92;&#093;\" title=\"Rendered by QuickLaTeX.com\"\/><\/p>We can obtain an approximate solution to the least-squares problem by replacing <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-841e32306afba64e7ef539a005755747_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#65;&#94;&#92;&#116;&#111;&#112;&#32;&#65;\" title=\"Rendered by QuickLaTeX.com\" height=\"15\" width=\"38\" style=\"vertical-align: 0px;\"\/> by <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-e2d016e91566b4ec17ebd357dbe265ea_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#82;&#94;&#92;&#116;&#111;&#112;&#32;&#82;\" title=\"Rendered by QuickLaTeX.com\" height=\"15\" width=\"39\" style=\"vertical-align: 0px;\"\/> in (5) and solving. The resulting solution is<p class=\"ql-center-displayed-equation\" style=\"line-height: 22px;\"><span class=\"ql-right-eqno\"> &nbsp; <\/span><span class=\"ql-left-eqno\"> &nbsp; <\/span><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-6b63093ea5c1d9714fb0deab43fc4dc3_l3.png\" height=\"22\" width=\"174\" class=\"ql-img-displayed-equation quicklatex-auto-format\" alt=\"&#92;&#091;&#120;&#95;&#48;&#32;&#61;&#32;&#82;&#94;&#123;&#45;&#49;&#125;&#32;&#40;&#82;&#94;&#123;&#45;&#92;&#116;&#111;&#112;&#125;&#40;&#65;&#94;&#92;&#116;&#111;&#112;&#32;&#98;&#41;&#41;&#46;&#92;&#093;\" title=\"Rendered by QuickLaTeX.com\"\/><\/p><\/p>\n\n\n\n<p>This solution <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-1cb654a3bb05fcd9620c19c83d513488_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#120;&#95;&#48;\" title=\"Rendered by QuickLaTeX.com\" height=\"11\" width=\"17\" style=\"vertical-align: -3px;\"\/> will typically not be a good solution to the least-squares problem (2), so we need to<em> <\/em>iterate. To do so, we\u2019ll try and solve for the error <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-054ff6e52b5fd40ff818d8b4de5b2f9b_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#120;&#32;&#45;&#32;&#120;&#95;&#48;\" title=\"Rendered by QuickLaTeX.com\" height=\"11\" width=\"49\" style=\"vertical-align: -3px;\"\/>. To derive an equation for the error, subtract <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-18f70a151f40dff5a0abb9d097c135df_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#65;&#94;&#92;&#116;&#111;&#112;&#32;&#65;&#32;&#120;&#95;&#48;\" title=\"Rendered by QuickLaTeX.com\" height=\"18\" width=\"55\" style=\"vertical-align: -3px;\"\/> from both sides of the normal equations (5), yielding <p class=\"ql-center-displayed-equation\" style=\"line-height: 22px;\"><span class=\"ql-right-eqno\"> &nbsp; <\/span><span class=\"ql-left-eqno\"> &nbsp; <\/span><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-4f1ee993844dbcd8e397f27ccd1deefb_l3.png\" height=\"22\" width=\"241\" class=\"ql-img-displayed-equation quicklatex-auto-format\" alt=\"&#92;&#091;&#40;&#65;&#94;&#92;&#116;&#111;&#112;&#32;&#65;&#41;&#40;&#120;&#45;&#120;&#95;&#48;&#41;&#32;&#61;&#32;&#65;&#94;&#92;&#116;&#111;&#112;&#32;&#40;&#98;&#45;&#65;&#120;&#95;&#48;&#41;&#46;&#92;&#093;\" title=\"Rendered by QuickLaTeX.com\"\/><\/p>Now, to solve for the error, substitute <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-e2d016e91566b4ec17ebd357dbe265ea_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#82;&#94;&#92;&#116;&#111;&#112;&#32;&#82;\" title=\"Rendered by QuickLaTeX.com\" height=\"15\" width=\"39\" style=\"vertical-align: 0px;\"\/> for <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-841e32306afba64e7ef539a005755747_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#65;&#94;&#92;&#116;&#111;&#112;&#32;&#65;\" title=\"Rendered by QuickLaTeX.com\" height=\"15\" width=\"38\" style=\"vertical-align: 0px;\"\/> again and solve for <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-b312d649591164b7149ed0756f694a76_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#120;\" title=\"Rendered by QuickLaTeX.com\" height=\"8\" width=\"10\" style=\"vertical-align: 0px;\"\/>, obtaining a new approximate solution <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-e08a13fffeaa35c71a527c0bfe1ee065_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#120;&#95;&#49;\" title=\"Rendered by QuickLaTeX.com\" height=\"11\" width=\"16\" style=\"vertical-align: -3px;\"\/>: <p class=\"ql-center-displayed-equation\" style=\"line-height: 22px;\"><span class=\"ql-right-eqno\"> &nbsp; <\/span><span class=\"ql-left-eqno\"> &nbsp; <\/span><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-0496d7e3c8bc117e658c57fc167081be_l3.png\" height=\"22\" width=\"317\" class=\"ql-img-displayed-equation quicklatex-auto-format\" alt=\"&#92;&#091;&#120;&#92;&#97;&#112;&#112;&#114;&#111;&#120;&#32;&#120;&#95;&#49;&#32;&#92;&#99;&#111;&#108;&#111;&#110;&#101;&#113;&#113;&#32;&#120;&#95;&#48;&#32;&#43;&#32;&#82;&#94;&#123;&#45;&#92;&#116;&#111;&#112;&#125;&#40;&#82;&#94;&#123;&#45;&#49;&#125;&#40;&#65;&#94;&#92;&#116;&#111;&#112;&#40;&#98;&#45;&#65;&#120;&#95;&#48;&#41;&#41;&#41;&#46;&#92;&#093;\" title=\"Rendered by QuickLaTeX.com\"\/><\/p><\/p>\n\n\n\n<p>We can now go another step: Derive an equation for the error <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-2d24e89258882d3539d15666e91474f5_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#120;&#45;&#120;&#95;&#49;\" title=\"Rendered by QuickLaTeX.com\" height=\"11\" width=\"48\" style=\"vertical-align: -3px;\"\/>, approximate <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-579a04eb83e4a817dbc74c286f1a3b58_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#65;&#94;&#92;&#116;&#111;&#112;&#32;&#65;&#32;&#92;&#97;&#112;&#112;&#114;&#111;&#120;&#32;&#82;&#94;&#92;&#116;&#111;&#112;&#32;&#82;\" title=\"Rendered by QuickLaTeX.com\" height=\"15\" width=\"101\" style=\"vertical-align: 0px;\"\/>, and obtain a new approximate solution <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-d525edc5789c75ed5956bf51c0d19e17_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#120;&#95;&#50;\" title=\"Rendered by QuickLaTeX.com\" height=\"11\" width=\"17\" style=\"vertical-align: -3px;\"\/>. Continuing this process, we obtain an iteration<p class=\"ql-center-displayed-equation\" style=\"line-height: 22px;\"><span class=\"ql-right-eqno\"> (6) <\/span><span class=\"ql-left-eqno\"> &nbsp; <\/span><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-f8db238fc978e6a8fc968fc5e387ec27_l3.png\" height=\"22\" width=\"291\" class=\"ql-img-displayed-equation quicklatex-auto-format\" alt=\"&#92;&#091;&#120;&#95;&#123;&#105;&#43;&#49;&#125;&#32;&#61;&#32;&#120;&#95;&#105;&#32;&#43;&#32;&#82;&#94;&#123;&#45;&#92;&#116;&#111;&#112;&#125;&#40;&#82;&#94;&#123;&#45;&#49;&#125;&#40;&#65;&#94;&#92;&#116;&#111;&#112;&#40;&#98;&#45;&#65;&#120;&#95;&#105;&#41;&#41;&#41;&#46;&#92;&#093;\" title=\"Rendered by QuickLaTeX.com\"\/><\/p>This iteration is known as the <em><a href=\"https:\/\/ar5iv.labs.arxiv.org\/html\/2002.01387#S10.SS4\">iterative sketching<\/a><\/em> method.<sup class=\"modern-footnotes-footnote \" data-mfn=\"8\" data-mfn-post-scope=\"000000000000057f0000000000000000_1632\"><a href=\"javascript:void(0)\"  role=\"button\" aria-pressed=\"false\" aria-describedby=\"mfn-content-000000000000057f0000000000000000_1632-8\">8<\/a><\/sup><span id=\"mfn-content-000000000000057f0000000000000000_1632-8\" role=\"tooltip\" class=\"modern-footnotes-footnote__note\" tabindex=\"0\" data-mfn=\"8\">The name <em>iterative sketching<\/em> is for historical reasons. <a href=\"https:\/\/arxiv.org\/abs\/1411.0347\">Original versions of the procedure<\/a> involved taking a fresh sketch <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-878253dd54a001bd6043d03c94e8c58a_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#83;&#95;&#105;&#65;&#32;&#61;&#32;&#81;&#95;&#105;&#82;&#95;&#105;\" title=\"Rendered by QuickLaTeX.com\" height=\"17\" width=\"91\" style=\"vertical-align: -4px;\"\/> at every iteration. Later, <a href=\"https:\/\/web.stanford.edu\/~pilanci\/papers\/IHSMomentum18.pdf\">it<\/a> <a href=\"https:\/\/arxiv.org\/abs\/1911.02675\">was<\/a> <a href=\"https:\/\/tropp.caltech.edu\/notes\/Tro20-Randomized-Algorithms-LN.pdf#page83\">realized<\/a> that a single sketch <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-f6b6ffdfa7888fa001ee9e26544c7384_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#83;&#65;\" title=\"Rendered by QuickLaTeX.com\" height=\"13\" width=\"25\" style=\"vertical-align: 0px;\"\/> suffices, albeit with a slower convergence rate. Typically, only having to sketch and QR factorize once is worth the slower convergence rate.<\/span> If the distortion is small enough, this method converges at an exponential rate, yielding a high-accuracy least squares solution after a few iterations.<\/p>\n\n\n\n<p>Let&#8217;s apply iterative sketching to the example we considered above. We show the forward error of the sketch-and-solve and direct methods as horizontal dashed purple and red lines. Iterative sketching begins at roughly the forward error of sketch-and-solve, with the error decreasing at an exponential rate until it reaches that of the direct method over the course of fourteen iterations. For this problem, at least, iterative sketching gives high-accuracy solutions to the least-squares problem!<\/p>\n\n\n<div class=\"wp-block-image\">\n<figure class=\"aligncenter size-large is-resized\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/uploads\/2023\/11\/iterative-sketching-1024x768.png\" alt=\"\" class=\"wp-image-1651\" width=\"768\" height=\"576\" srcset=\"https:\/\/www.ethanepperly.com\/wp-content\/uploads\/2023\/11\/iterative-sketching-1024x768.png 1024w, https:\/\/www.ethanepperly.com\/wp-content\/uploads\/2023\/11\/iterative-sketching-300x225.png 300w, https:\/\/www.ethanepperly.com\/wp-content\/uploads\/2023\/11\/iterative-sketching-768x576.png 768w, https:\/\/www.ethanepperly.com\/wp-content\/uploads\/2023\/11\/iterative-sketching.png 1120w\" sizes=\"auto, (max-width: 768px) 100vw, 768px\" \/><\/figure><\/div>\n\n\n<p>To summarize, we&#8217;ve now seen two very different ways of using sketching:<\/p>\n\n\n\n<ul class=\"wp-block-list\"><li><strong>Sketch-and-solve.<\/strong> Sketch the data <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-11f4f587954b361e7d78940f65b8d70d_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#65;\" title=\"Rendered by QuickLaTeX.com\" height=\"13\" width=\"13\" style=\"vertical-align: 0px;\"\/> and <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-3ab6f6c64ce17f786a81f8cc8fdcec90_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#98;\" title=\"Rendered by QuickLaTeX.com\" height=\"12\" width=\"8\" style=\"vertical-align: 0px;\"\/> and solve the sketched least-squares problem (3). The resulting solution <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-aaab1105f322d56886350f3bdc22c2b8_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#92;&#104;&#97;&#116;&#123;&#120;&#125;\" title=\"Rendered by QuickLaTeX.com\" height=\"14\" width=\"11\" style=\"vertical-align: 0px;\"\/> is cheap to obtain, but may have low accuracy.<\/li><li><strong>Iterative sketching.<\/strong> Sketch the matrix <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-11f4f587954b361e7d78940f65b8d70d_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#65;\" title=\"Rendered by QuickLaTeX.com\" height=\"13\" width=\"13\" style=\"vertical-align: 0px;\"\/> and obtain an approximation <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-76cbbe605ab854931d8785ea734db4a6_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#82;&#94;&#92;&#116;&#111;&#112;&#32;&#82;&#32;&#61;&#32;&#40;&#83;&#65;&#41;&#94;&#92;&#116;&#111;&#112;&#32;&#40;&#83;&#65;&#41;\" title=\"Rendered by QuickLaTeX.com\" height=\"20\" width=\"151\" style=\"vertical-align: -5px;\"\/> to <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-841e32306afba64e7ef539a005755747_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#65;&#94;&#92;&#116;&#111;&#112;&#32;&#65;\" title=\"Rendered by QuickLaTeX.com\" height=\"15\" width=\"38\" style=\"vertical-align: 0px;\"\/>. Use the approximation <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-e2d016e91566b4ec17ebd357dbe265ea_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#82;&#94;&#92;&#116;&#111;&#112;&#32;&#82;\" title=\"Rendered by QuickLaTeX.com\" height=\"15\" width=\"39\" style=\"vertical-align: 0px;\"\/> to produce a sequence of better-and-better least-squares solutions <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-44092f757392a8bd31d874dabe451fd3_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#120;&#95;&#105;\" title=\"Rendered by QuickLaTeX.com\" height=\"11\" width=\"15\" style=\"vertical-align: -3px;\"\/> by the iteration (6). If we run for enough iterations <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-1d83a8e8f99aa0fb16ebf30cc43326a2_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#113;\" title=\"Rendered by QuickLaTeX.com\" height=\"12\" width=\"8\" style=\"vertical-align: -4px;\"\/>, the accuracy of the iterative sketching solution <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-1ccb015633bf7fd22aca592e0b21f7fe_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#120;&#95;&#113;\" title=\"Rendered by QuickLaTeX.com\" height=\"14\" width=\"17\" style=\"vertical-align: -6px;\"\/> can be quite high.<\/li><\/ul>\n\n\n\n<p>By combining sketching with more sophisticated iterative methods such as <a href=\"https:\/\/www.cs.cmu.edu\/~quake-papers\/painless-conjugate-gradient.pdf\">conjugate gradient<\/a> and <a href=\"https:\/\/stanford.edu\/group\/SOL\/software\/lsqr\/lsqr-toms82a.pdf\">LSQR<\/a>, we can get an even faster-converging least-squares algorithm, known as <em><a rel=\"noreferrer noopener\" href=\"https:\/\/ar5iv.labs.arxiv.org\/html\/2002.01387#S10.SS5\" target=\"_blank\">sketch-and-precondition<\/a><\/em>. Here&#8217;s the same plot from above with sketch-and-precondition added; we see that sketch-and-precondition converges even faster than iterative sketching does!<\/p>\n\n\n<div class=\"wp-block-image\">\n<figure class=\"aligncenter size-large is-resized\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/uploads\/2023\/11\/sketch-and-precondition-1024x768.png\" alt=\"\" class=\"wp-image-1652\" width=\"768\" height=\"576\" srcset=\"https:\/\/www.ethanepperly.com\/wp-content\/uploads\/2023\/11\/sketch-and-precondition-1024x768.png 1024w, https:\/\/www.ethanepperly.com\/wp-content\/uploads\/2023\/11\/sketch-and-precondition-300x225.png 300w, https:\/\/www.ethanepperly.com\/wp-content\/uploads\/2023\/11\/sketch-and-precondition-768x576.png 768w, https:\/\/www.ethanepperly.com\/wp-content\/uploads\/2023\/11\/sketch-and-precondition.png 1120w\" sizes=\"auto, (max-width: 768px) 100vw, 768px\" \/><\/figure><\/div>\n\n\n<p>&#8220;Does sketching work?&#8221; Even for a simple problem like least-squares, the answer is complicated:<\/p>\n\n\n\n<blockquote class=\"wp-block-quote is-layout-flow wp-block-quote-is-layout-flow\"><p>A direct use of sketching (i.e., sketch-and-solve) leads to a fast, low-accuracy solution to least-squares problems. But sketching can achieve much higher accuracy for least-squares problems by combining sketching with an iterative method (iterative sketching and sketch-and-precondition).<\/p><\/blockquote>\n\n\n\n<p>We&#8217;ve focused on least-squares problems in this section, but these conclusions could hold more generally. If &#8220;sketching doesn&#8217;t work&#8221; in your application, maybe it would if it was combined with an iterative method.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Just How Accurate Can Sketching Be?<\/h2>\n\n\n\n<p>We left our discussion of sketching-plus-iterative-methods in the previous section on a positive note, but there is one last lingering question that remains to be answered. We stated that iterative sketching (and sketch-and-precondition) converge at an exponential rate. But our computers <a href=\"https:\/\/en.m.wikipedia.org\/wiki\/Floating-point_arithmetic\">store numbers to only so much precision<\/a>; in practice, the accuracy of an iterative method has to saturate at some point.<\/p>\n\n\n\n<p>An (iterative) least-squares solver is said to be <em><a href=\"https:\/\/nhigham.com\/2020\/08\/04\/what-is-numerical-stability\/\">forward stable<\/a><\/em> if, when run for a sufficient number <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-1d83a8e8f99aa0fb16ebf30cc43326a2_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#113;\" title=\"Rendered by QuickLaTeX.com\" height=\"12\" width=\"8\" style=\"vertical-align: -4px;\"\/> of iterations, the final accuracy <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-985c56a5188551599a4f47228ea9c472_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#92;&#110;&#111;&#114;&#109;&#123;&#120;&#95;&#113;&#32;&#45;&#32;&#120;&#95;&#92;&#115;&#116;&#97;&#114;&#125;\" title=\"Rendered by QuickLaTeX.com\" height=\"20\" width=\"71\" style=\"vertical-align: -6px;\"\/> is comparable to accuracy of a standard <em><a href=\"https:\/\/en.m.wikipedia.org\/wiki\/QR_decomposition\">direct method<\/a><\/em> for the least-squares problem like MATLAB&#8217;s <a href=\"https:\/\/www.mathworks.com\/help\/matlab\/ref\/mldivide.html\">\\ command<\/a> or Python&#8217;s <em><a href=\"https:\/\/docs.scipy.org\/doc\/scipy\/reference\/generated\/scipy.linalg.lstsq.html\">scipy.linalg.lstsq<\/a><\/em>. Forward stability is not about <em>speed<\/em> or <em>rate of convergence<\/em> but about the <em>maximum achievable accuracy<\/em>.<\/p>\n\n\n\n<p>The stability of sketch-and-precondition was studied in a <a href=\"https:\/\/arxiv.org\/abs\/2302.07202\">recent paper<\/a> by <a href=\"https:\/\/www.maths.ox.ac.uk\/people\/maike.meier\">Meier<\/a>, <a href=\"https:\/\/people.maths.ox.ac.uk\/nakatsukasa\/\">Nakatsukasa<\/a>, <a href=\"https:\/\/math.cornell.edu\/alex-townsend\">Townsend<\/a>, and <a href=\"https:\/\/personalpages.manchester.ac.uk\/staff\/marcus.webb\/\">Webb<\/a>. They demonstrated that, with the initial iterate <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-582b045cb1264dacb4c6888a6b5114fe_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#120;&#95;&#48;&#32;&#61;&#32;&#48;\" title=\"Rendered by QuickLaTeX.com\" height=\"15\" width=\"50\" style=\"vertical-align: -3px;\"\/>, sketch-and-precondition is not forward stable. The maximum achievable accuracy was worse than standard solvers by orders of magnitude! Maybe sketching doesn&#8217;t work after all?<\/p>\n\n\n\n<p>Fortunately, there is good news:<\/p>\n\n\n\n<ul class=\"wp-block-list\"><li>The iterative sketching method is provably forward stable. This result is shown in my <a href=\"https:\/\/arxiv.org\/abs\/2311.04362\">newly released paper<\/a>; check it out if you&#8217;re interested!<\/li><li>If we use the sketch-and-solve method <em>as the initial iterate <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-e2d3d45e407ce3ee2c0463ed3655942b_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#120;&#95;&#48;&#32;&#61;&#32;&#92;&#104;&#97;&#116;&#123;&#120;&#125;\" title=\"Rendered by QuickLaTeX.com\" height=\"17\" width=\"52\" style=\"vertical-align: -3px;\"\/> for sketch-and-precondition<\/em>, then sketch-and-precondition appears to be forward stable in practice. No theoretical analysis supporting this finding is known at present.<sup class=\"modern-footnotes-footnote \" data-mfn=\"9\" data-mfn-post-scope=\"000000000000057f0000000000000000_1632\"><a href=\"javascript:void(0)\"  role=\"button\" aria-pressed=\"false\" aria-describedby=\"mfn-content-000000000000057f0000000000000000_1632-9\">9<\/a><\/sup><span id=\"mfn-content-000000000000057f0000000000000000_1632-9\" role=\"tooltip\" class=\"modern-footnotes-footnote__note\" tabindex=\"0\" data-mfn=\"9\">For those interested, neither iterative sketching nor sketch-and-precondition are <a href=\"https:\/\/nhigham.com\/2020\/08\/04\/what-is-numerical-stability\/\">backward stable<\/a>, which is a stronger stability guarantee than forward stability. Fortunately, forward stability is a perfectly adequate stability guarantee for many\u2014but not all\u2014applications.<\/span><\/li><\/ul>\n\n\n\n<p>These conclusions are pretty nuanced. To see what&#8217;s going, it can be helpful to look at a graph:<sup class=\"modern-footnotes-footnote \" data-mfn=\"10\" data-mfn-post-scope=\"000000000000057f0000000000000000_1632\"><a href=\"javascript:void(0)\"  role=\"button\" aria-pressed=\"false\" aria-describedby=\"mfn-content-000000000000057f0000000000000000_1632-10\">10<\/a><\/sup><span id=\"mfn-content-000000000000057f0000000000000000_1632-10\" role=\"tooltip\" class=\"modern-footnotes-footnote__note\" tabindex=\"0\" data-mfn=\"10\">For another randomly generated least-squares problem of the same size with condition number <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-249974e9137aecdf69a8846448a5a474_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#49;&#48;&#94;&#123;&#49;&#48;&#125;\" title=\"Rendered by QuickLaTeX.com\" height=\"15\" width=\"31\" style=\"vertical-align: 0px;\"\/> and residual <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-5019e82d2dc633a466a8b2c8f3b0ef69_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#49;&#48;&#94;&#123;&#45;&#54;&#125;\" title=\"Rendered by QuickLaTeX.com\" height=\"15\" width=\"35\" style=\"vertical-align: 0px;\"\/>.<\/span>\n\n\n<div class=\"wp-block-image\">\n<figure class=\"aligncenter size-large is-resized\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/uploads\/2023\/11\/bad-sketch-and-precondition-1024x768.png\" alt=\"\" class=\"wp-image-1660\" width=\"768\" height=\"576\" srcset=\"https:\/\/www.ethanepperly.com\/wp-content\/uploads\/2023\/11\/bad-sketch-and-precondition-1024x768.png 1024w, https:\/\/www.ethanepperly.com\/wp-content\/uploads\/2023\/11\/bad-sketch-and-precondition-300x225.png 300w, https:\/\/www.ethanepperly.com\/wp-content\/uploads\/2023\/11\/bad-sketch-and-precondition-768x576.png 768w, https:\/\/www.ethanepperly.com\/wp-content\/uploads\/2023\/11\/bad-sketch-and-precondition.png 1120w\" sizes=\"auto, (max-width: 768px) 100vw, 768px\" \/><\/figure><\/div>\n\n\n<p>The performance of different methods can be summarized as follows: Sketch-and-solve can have very poor forward error. Sketch-and-precondition with the zero initialization <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-582b045cb1264dacb4c6888a6b5114fe_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#120;&#95;&#48;&#32;&#61;&#32;&#48;\" title=\"Rendered by QuickLaTeX.com\" height=\"15\" width=\"50\" style=\"vertical-align: -3px;\"\/> is better, but still much worse than the direct method. Iterative sketching and sketch-and-precondition with <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.ethanepperly.com\/wp-content\/ql-cache\/quicklatex.com-e2d3d45e407ce3ee2c0463ed3655942b_l3.png\" class=\"ql-img-inline-formula quicklatex-auto-format\" alt=\"&#120;&#95;&#48;&#32;&#61;&#32;&#92;&#104;&#97;&#116;&#123;&#120;&#125;\" title=\"Rendered by QuickLaTeX.com\" height=\"17\" width=\"52\" style=\"vertical-align: -3px;\"\/> fair much better, eventually achieving an accuracy comparable to the direct method.<\/p>\n\n\n\n<p>Put more simply, appropriately implemented, sketching works after all!<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Conclusion<\/h2>\n\n\n\n<p>Sketching is a computational tool, just like the <a href=\"https:\/\/www.ethanepperly.com\/index.php\/2021\/05\/10\/big-ideas-in-applied-math-the-fast-fourier-transform\/\">fast Fourier transform<\/a> or the <a href=\"https:\/\/www.ethanepperly.com\/index.php\/2023\/06\/12\/low-rank-approximation-toolbox-randomized-svd\/\">randomized SVD<\/a>. Sketching can be used effectively to solve some problems. But, like any computational tool, sketching is not a silver bullet. Sketching allows you to dimensionality-reduce matrices and vectors, but it distorts them by an appreciable amount. Whether or not this distortion is something you can live with depends on your problem (how much accuracy do you need?) and how you use the sketch (sketch-and-solve or with an iterative method).<\/p>\n","protected":false},"excerpt":{"rendered":"<p>I&#8217;m excited to share that my paper, Fast and forward stable randomized algorithms for linear least-squares problems has been released as a preprint on arXiv. With the release of this paper, now seemed like a great time to discuss a topic I\u2019ve been wanting to write about for a while: sketching. For the past two<a class=\"more-link\" href=\"https:\/\/www.ethanepperly.com\/index.php\/2023\/11\/13\/does-sketching-work\/\">Read more<\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[6,7,12],"tags":[],"class_list":["post-1632","post","type-post","status-publish","format-standard","hentry","category-expository","category-research","category-sketching"],"_links":{"self":[{"href":"https:\/\/www.ethanepperly.com\/index.php\/wp-json\/wp\/v2\/posts\/1632","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.ethanepperly.com\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.ethanepperly.com\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.ethanepperly.com\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.ethanepperly.com\/index.php\/wp-json\/wp\/v2\/comments?post=1632"}],"version-history":[{"count":24,"href":"https:\/\/www.ethanepperly.com\/index.php\/wp-json\/wp\/v2\/posts\/1632\/revisions"}],"predecessor-version":[{"id":1669,"href":"https:\/\/www.ethanepperly.com\/index.php\/wp-json\/wp\/v2\/posts\/1632\/revisions\/1669"}],"wp:attachment":[{"href":"https:\/\/www.ethanepperly.com\/index.php\/wp-json\/wp\/v2\/media?parent=1632"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.ethanepperly.com\/index.php\/wp-json\/wp\/v2\/categories?post=1632"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.ethanepperly.com\/index.php\/wp-json\/wp\/v2\/tags?post=1632"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}