\documentclass[11pt]{article}
\usepackage{amsmath,amssymb,amsthm}

\DeclareMathOperator*{\E}{\mathbb{E}}
\let\Pr\relax
\DeclareMathOperator*{\Pr}{\mathbb{P}}
\DeclareMathOperator*{\In}{\mathbb{I}}
\DeclareMathOperator*{\Fo}{\mathcal{F}}
\DeclareMathOperator*{\median}{median}

\newcommand{\eps}{\epsilon}
\newcommand{\inprod}[1]{\left\langle #1 \right\rangle}
\newcommand{\R}{\mathbb{R}}
\newcommand{\poly}{{\rm poly}}


\newcommand{\handout}[5]{
  \noindent
  \begin{center}
  \framebox{
    \vbox{
      \hbox to 5.78in { {\bf CS 395T: Sublinear Algorithms } \hfill #2 }
      \vspace{4mm}
      \hbox to 5.78in { {\Large \hfill #5  \hfill} }
      \vspace{2mm}
      \hbox to 5.78in { {\em #3 \hfill #4} }
    }
  }
  \end{center}
  \vspace*{4mm}
}

\newcommand{\lecture}[4]{\handout{#1}{#2}{#3}{Scribe: #4}{Lecture #1 }}

\newtheorem{thm}{Theorem}[section]
\newtheorem{claim}[thm]{Claim}
\newtheorem{lemma}[thm]{Lemma}
\newtheorem{define}[thm]{Definition}
\newtheorem{cor}[thm]{Corollary}
\newtheorem{obs}[thm]{Observation}
\newtheorem{remark}[thm]{Remark}
\newtheorem{example}[thm]{Example}
\newtheorem{construction}[thm]{Construction}
\newtheorem{conjecture}{Conjecture}
\newtheorem{THM}{Theorem}
\newtheorem{COR}[THM]{Corollary}
\newtheorem{question}{Question}
\newtheorem{fact}[thm]{Fact}
\newtheorem{prop}[thm]{Proposition}
% 1-inch margins, from fullpage.sty by H.Partl, Version 2, Dec. 15, 1988.
\topmargin 0pt
\advance \topmargin by -\headheight
\advance \topmargin by -\headsep
\textheight 8.9in
\oddsidemargin 0pt
\evensidemargin \oddsidemargin
\marginparwidth 0.5in
\textwidth 6.5in

\parindent 0in
\parskip 1.5ex

\begin{document}

\lecture{13 --- Oct. 9, 2014}{Fall 2014}{Prof.\ Eric Price}{Udit Agarwal}

In today's lecture, we will cover the following topics:

\begin{enumerate}
\item $l_2/l_1$ upper bounds \cite{Candes2006}

\item $l_2/l_2$ lower bounds \cite{Cohen2009} :- To perform $l_2/l_2$ recovery deterministically, at least $\Omega(n)$ samples are required.

\item $l_1/l_1$ lower bounds \cite{Price2010} :- To perform $l_1/l_1$ recovery deterministically or with a randomized algorithm, at least $\Omega(k\log \frac{n}{k})$ samples are required.
\end{enumerate}


Consider the problem of stable sparse recovery: given a matrix $A \in \R^{m \times n}$ and a $k$-sparse vector $x$ and given $y = Ax + e$, with $e$ as the error term, we wish to recover $\hat{x}$ such that $$|| \hat{x} -x ||_{2}  \leq C||e||_2$$.

\textbf{Ques: } What about if $x$ is not $k$-sparse??

Then the problem becomes: given $Ax$, $\forall x \in \R^n$, recover $\hat{x}$ such that $$||\hat{x} - x||_p \leq C \underset{\text{$k$-sparse $x'$}}{\min} 
	||x-x'||_q $$
for some norm parameters $p$ and $q$ and an approximation factor $C$.
 
Thus, the error term depends only on the top $k$ terms of $x$. Some of the $l_p/l_q$ recovery guarantees are as follows:

\begin{itemize}
\item $l_2/l_2$ : $$ ||\hat{x} - x||_2 \leq C||x - x_k||_2 $$

\item $l_2/l_1$ : $$ ||\hat{x} - x||_2 \leq \frac{C}{\sqrt{k}}||x - x_k||_1$$

\item $l_1/l_1$ : $$ ||\hat{x} - x||_1 \leq C||x - x_k||_1$$
\end{itemize}

where $x_k$ contains the top $k$ terms of $x$.

Now, we'll talk about the bounds on the number of samples required to perform each of these $l_p/l_q$ guarantees deterministically:

\begin{itemize}
\item $l_2/l_2$ : To perform $l_2/l_2$ recovery deterministically, at least $\Omega(n)$ samples are required.

\item $l_2/l_1$ : To perform $l_2/l_1$ recovery deterministically, at least $O(k\log \frac{n}{k})$ samples are required. 

\item $l_1/l_1$ : To perform $l_1/l_1$ recovery deterministically, at least $\Omega(k\log \frac{n}{k})$ samples are required and can be done in $O(n\log n)$ time. 
\end{itemize}

\section{$l_2/l_1$ Recovery Upper Bound}

We are given a matrix $A \in \R^{m \times n}$ that satisfies RIP and $Y = Ax_{2k} + e$, where $e$ is the error term. Then, we have 
\begin{equation} \label{eq:eq1}
||\hat{x} - x_k|| \leq C||e||_2
\end{equation}

\textbf{Ques: } What about if $x$ is non-sparse??

We have, $Ax = Ax_{2k} + A(x - x_{2k})$. 

From \eqref{eq:eq1}, running with $k' = 2k$, we get that
\begin{equation}\label{eq:eq2}
||\hat{x} - x_{2k}||_2 \leq C||A(x - x_{2k})||_2
\end{equation} 

Now, we'll use a shelling argument, similar to one we described in the
last class.

Split $x$ into blocks $x^{(1)}, x^{(2)},...$ of size $k$, so that $x^{(1)}$ has the largest $k$ coordinates, and each next $x^{(i)}$ has the next largest $k$ coordinates. Then, we have $$x - x_{2k} = x^{(3)} + x^{(4)} + ...$$

Then, 
\begin{align*}\\
||A(x - x_{2k})||_2 & = ||A.\sum_{i=3}^{\infty} x^{(i)}||_2	 \\
& \leq \sum_{i=3}^{\infty} ||Ax^{(i)}||_2 \\
& \leq \sum_{i=3}^{\infty} (1 + \epsilon)||x^{(i)}||_2 && \text{(As $A$ satisfies the RIP)} \\
& \leq (1 + \epsilon) \sum_{i=3}^{\infty} \sqrt{k}||x^{(i)}||_{\infty} \\
& \leq (1 + \epsilon) \sum_{i=3}^{\infty} \sqrt{k}\frac{||x^{(i-1)}||_1}{k} \\
& = \frac{(1 + \epsilon)}{\sqrt{k}} \sum_{i=2}^{\infty} ||x^{(i)}||_1 \\
& = \frac{(1 + \epsilon)}{\sqrt{k}} ||x - x_k||_1 \\
\end{align*}

Now, plugging this inequality in equation \eqref{eq:eq2}, we have 
\begin{equation}		\label{eq:eq3}
||\hat{x} - x_k||_2 \leq C\frac{(1 + \epsilon)}{\sqrt{k}} ||x - x_k||_1
\end{equation}

Now, we have 
\begin{equation} \label{eq:eq4}
||\hat{x} - x||_2 \leq ||\hat{x} - x_{2k}||_2 + ||x - x_{2k}||_2
\end{equation}

Also, by plugging $A = I$ in the previous argument, we have 
\begin{equation}		\label{eq:eq5}
||x - x_{2k}||_2 \leq \frac{1}{\sqrt{k}}||x - x_k||_1
\end{equation}

Now using equations \eqref{eq:eq4} \& \eqref{eq:eq5} in equation \eqref{eq:eq3}, we have $$ ||\hat{x} - x_k||_2 \leq (C\frac{(1 + \epsilon)}{\sqrt{k}} + \frac{1}{\sqrt{k}}) ||x - x_k||_1$$

\section{$l_1/l_1$ Recovery Algorithm}

We have seen in Problem 2 of Problem Set 2 that $(k, C/\sqrt{k})$ $l_2/l_1$ recovery implies $(k, O(C))$ $l_1/l_1$ recovery. Hence, $l_1/l_1$ recovery guarantee is taken care of by the results in the previous section.

\section{$l_2/l_2$ Recovery: Deterministic Lower Bound}

We will show that deterministic $l_2/l_2$ recovery requires
$\Omega(n)$ samples even for $k=1$.  So let's think about the $k=1$
case.

Now, suppose we are given $ y = Ax$ for some $A \in \R^{m \times n}$
and $x \in \R^n$, and can recover $\hat{x}$ such that $$||\hat{x} -
x||_2 \leq C \min_{i \in [n]} ||x - x_i||_2$$ where $x_i$ contains the
top $i$ terms of $x$.


If $y = 0$, then $\hat{x}$ must also be zero vector.

Thus, $\forall x \in \mathcal{N} := nullspace(A)$, we need $0$ to be an OK output.

Then $\forall j \in [n]$ and $x \in \mathcal{N}$,
\begin{align*}
\sum_{i} x_i^2 & \leq C^2 \sum_{i \neq j} x_i^2 \\
x_j^2 & \leq (C^2 - 1).(||x||_2^2 - x_j^2)
\end{align*}
\begin{align}\label{eq:alpha}
  x_j^2 \leq \underbrace{(1 - \frac{1}{C^2})}_{\text{$\alpha < 1$}}.||x||_2^2
\end{align}

Our goal is to show that~\eqref{eq:alpha} implies that the dimension
of $\mathcal{N}$ must be small.

Let $v_1,\dots,v_{n-m}$ be the orthogonal basis for
$\mathcal{N}$. Thus, $(n-m)$ is the dimension of the null space
$\mathcal{N}$.

Let $e_i \in \R^{n}$ such that it's $i$-th entry is $1$ and the rest of the entries are $0$.
 
Then, $Proj_{\mathcal{N}}(e_i)$ (the orthogonal projection of $e_i$ onto $\mathcal{N}$) = $\sum_{j=1}^{n-m} v_{j}v_{j}^{T}e_i$ 

Since $Proj_{\mathcal{N}}(e_i) \in \mathcal{N}$,
using~\eqref{eq:alpha} we have for the $i$th coordinate that
\begin{align}
(Proj_{\mathcal{N}}(e_i))_i = \sum_{j=1}^{n-m} e_i^{T}v_jv_j^Te_i & \leq \sqrt{\alpha.||Proj_{\mathcal{N}}(e_i)||_2^2} \\
\sum_{j=1}^{n-m}|\langle v_j,e_i\rangle|^2 & \leq \sqrt{\alpha} \label{eq:eq6}	
\end{align}


Now, sum equation \eqref{eq:eq6} over $i \in \{1,\dots,n\}$ and find 
\begin{align*}
n - m & = \sum_{j=1}^{n-m} \|v_j\|_2^2\\
&= 
\sum_{j=1}^{n-m} \sum_{i=1}^{n} |\langle v_j,e_i\rangle|^2 \\
& \leq n\sqrt{\alpha} \\
& < (1 - \frac{1}{2C^2})n 
\end{align*}
using~\eqref{eq:alpha}.  Hence, $ m \geq \frac{n}{2C^2}$.

This was proved by Albert Cohen, Wolfgang Dahmen, and Ronald DeVore \cite{Cohen2009} .

\section{Deterministic $l_1/l_1$ lower bound \cite{Price2010}}

\textbf{Idea:} We need to find a large set of well-separated sparse points and we should be able to cover them even in presence of lot of noise. 

We'll use a Volume Argument to find such a set of points.

\subsection{Gilbert-Varshamov Bound} 

They showed that $\forall q,k \in \mathbb{Z}^{+}$, $\epsilon \in \R^+$
with $0 < \epsilon < 1-\frac{1}{q}$, $\exists$ a set $S \subseteq [q]^k$
such that $S$ has minimum Hamming Distance $\epsilon k$ and $$\log |S|
\geq (1 - H_{q}(\epsilon))k\log q$$ where $H_q$ is the $q$-ary entropy
function $$H_q(\epsilon) = -\epsilon \log_q (\frac{\epsilon}{q-1}) -
(1 - \epsilon)\log_q (1 - \epsilon)$$

If we set $q = \frac{n}{k}$ and $\epsilon = \frac{1}{2}$, then $S
\subseteq [\frac{n}{k}]^k$ and has minimum hamming distance equal to
$\frac{k}{2}$ and $\log |S| \gtrsim k\log \frac{n}{k}$.

We can transform the set $[q]^k$ to $\{0,1\}^k$ by taking each
character and writing it into a unit. For example, $$ 5 \rightarrow
(0,0,0,0,1,0,\dots,0)$$ $$6 \rightarrow (0,0,0,0,0,1,0,\dots,0)$$

This gives us a set $S \subseteq \{0,1\}^n$ consisting of only
$k$-sparse vectors with minimum $l_1$-distance $k$ and $\log |S|
\gtrsim k\log \frac{n}{k}$.

Now, suppose $x \in S$, $||w||_1 \leq \frac{k}{10}$, and we recover
$\hat{x}$ from $y = A(x + w)$.

We know that 
\begin{align*}
||\hat{x} - (x + w)||_1 & \leq 2 \underset{\text{$k$-sparse $x'$}}{\min} ||(x + w) - x'||_1 \\
& \leq 2 \cdot \frac{k}{10} \hspace{10 pt} \text{(can be achieved by plugging $x' = x$)} \\
& = \frac{k}{5} \\
\end{align*}   

Now, we have 
\begin{align*}
||\hat{x} - x||_1 & \leq ||w||_1 + ||\hat{x} - (x + w)||_1 \\
& \leq \frac{3}{10}k \\
& < \frac{k}{2} \\
\end{align*}

We have bunch of points $x \in S$ and $S \subset B_{1}.k$, where $B_1$ is the $l_1$ ball in $\R^n$.

Now, $\forall x_i \in S$, consider a ball $(x_i + \frac{k}{10}B_1)$. For any given real matrix $A \in \R^{m \times n}$, we can project the ball $(x_i + \frac{k}{10}B_1)$ to $A(x_i + \frac{k}{10}B_1)$ and these balls are disjoint for different $x_i \in S$. And as $\cup_{x_i \in S} (x_i + \frac{k}{10}B_1) \subset \frac{11}{10}kB_1$, all these projected balls lies inside $A(\frac{11}{10}kB_1)$. 

Now, the volume of each of the projected small balls is equal to $Vol(A(\frac{k}{10}B_1))$ and that of the bigger ball inside which each of the disjoint smaller balls lie is equal to $Vol(A(\frac{11}{10}kB_1))$. And, we have 
\begin{equation}	 \label{eq:eq7}
\frac{Vol(A(\frac{11}{10}kB_1))}{Vol(A(\frac{k}{10}B_1))} = 11^m
\end{equation} 

\textbf{Note: } $AB_1$ is some convex shape in $\R^m$.

As the smaller balls are disjoint and they lie inside the bigger ball, we have
\begin{align*}
|S|Vol(A(\frac{k}{10}B_1)) & \leq Vol(A(\frac{11}{10}kB_1)) \\
|S| & \leq 11^m \hspace{10 pt} \text{(from equation \eqref{eq:eq7})} \\
m & \geq \log_{11} |S| \\
m & \gtrsim k\log \frac{n}{k} \\
\end{align*}
 
\section{Randomized Lower Bound \cite{Price2010}}

We'll show that any matrix $A \in \R^{m \times n}$ which is used for randomized $l_1/l_1$ recovery must have at least $m = \Omega(k\log \frac{n}{k})$ rows. We'll first assume that each of the entries $A_{ij}$ is an integer with $O(\log n)$ bits. 

Thus, the vector $Ax$ requires $O(m\log n)$ bits. Thus, in total $\Omega(k\log \frac{n}{k}\log n)$ bits must be stored for $Ax$ where each $x_i$ is poly-precision ($\log n$ bits per entry). 

Let $S$ be a set of $k$-sparse binary vectors and has minimum hamming distance $k$ and $\log |S| \gtrsim k\log \frac{n}{k}$. 

Now, consider $x_1,x_2,\dots,x_R \in S$.

Let $$z = x_1 + \underbrace{\frac{1}{11}x_2 + \frac{1}{11^2}x_3 + \dots + \frac{1}{11^{R-1}}x_R}_{= w' \text{(let)}}$$  

We have, $$||w'|| \leq k(\frac{1}{11} + \frac{1}{11^2} + \dots) = \frac{k}{10}$$

Rounding the recovery $\wh{z}$ of $y = Az$ to $S$ gives $x_1$.

\textbf{Note: } We can relate this problem to a Communication
Complexity problem. Consider the following communication game. There
are two parties, Alice and Bob. Alice is given the $R$ vectors $x_1,
x_2, x_3, \dots, x_R$ from set $S$. Now, Alice sends the vector $Ax$
as a message to Bob, who must recover the vectors $x_1, x_2, x_3,
\dots, x_R$ from $Az$, which implies that $Az$ has indeed $\Omega(R
\log \abs{S}) = \Omega(Rk\log \frac{n}{k})$ bits.


Let 
\begin{align*}
y^{(2)} & = (y - Ax_1).11 \\
& = A.(x_2 + \frac{1}{11}x_3 + \frac{1}{11^2}x_4 + \dots) \\
\end{align*}

Now, rounding $y^{(2)}$ to $S$ gives us $x_2$.

We can adopt the same strategy to recover all other $x_i$'s for all $1 \leq i \leq R$.

If this algorithm works with probability $\geq 1 - \frac{1}{2R}$, then
probably all stages succeed and we can recover all the $x_i$, which is
$\Omega(Rk\log \frac{n}{k})$ bits.

If $A$ has $O(\log n)$ bits per coordinate, then $Az$ has $O(R + \log n)$
bits per coordinate.

If $R \geq \log n$, then this means we have communicated $\Omega(Rk
\log \frac{n}{k})$ bits of information using only $O(m \log n)$ bits
of transmission.  Hence
\begin{align*}
m\log n & \gtrsim k\log \frac{n}{k}\log n \\
m & \gtrsim k\log \frac{n}{k} \\
\end{align*} 


\subsection{Removing the assumptions}

The above proof had two flaws: it assumed that the entries of $A$ were
integers with $O(\log n)$ bits per entry, and it required the
algorithm to succeed with probability $1 - \frac{1}{2\log n}$
probability.  Neither of these is necessary to the proof.

To decrease the probability requirement, consider the following
communication game. There are two parties, Alice and Bob. Alice is
given a string $z \in \{0,1\}^n$. Bob is given an index $i \in [n]$,
together with $z_1, z_2, \dots, z_{i-1}$. Now Alice sends some message
to Bob, who must output $z_i$ with probability at least
$\frac{3}{4}$. We refer to this problem as Augmented Indexing.  It is
known that solving Augmented Indexing requires lots of communication:

\begin{thm}[\cite{BJKS}]
  Any protocol that solved Augmented Indexing requires $\Omega(n)$
  bits of communication.
\end{thm}


In our current setting, Alice has a bit string of length $R \log
\abs{S}$, which she converts into vectors $x_1, x_2, \dots, x_R \in
S$. Bob converts his inputs into an index $i \in [R]$ and vectors
$x_1, x_2, \dots, x_{i-1}$, and wants to learn $x_i$. Now Alice sends
the vector $Az$ to Bob, who must recover the vector $x_i$.  This he
does in the same manner as before, but now he \emph{knows} he has
$x_1, \dotsc, x_{i-1}$ correctly.  Thus we don't need the union bound,
and get a lower bound even for constant probabilities of success.

Finally, we briefly mention how to remove the requirement that $A$
have coordinates that are $O(\log n)$ bit integers.  We observe two
things:

First, we can assume that $A$ has orthonormal rows.  This is because
for any $A$ there exists an invertible transformation $B$ such that
$BA$ has orthonormal rows.  Then sparse recovery using $Ax$ is
possible if and only if sparse recovery using $BAx$ is possible.  So
it suffices to give a lower bound for $A$ with orthonormal rows.

Second, we show that if $A$ has orthonormal rows then rounding $A$ to
$O(\log n)$ bits per entry has a negligible effect on the results of
the algorithm.  We use the following lemma:

\begin{lemma} \cite{Price2010} Consider any $m \times n$ matrix $A$
  with orthonormal rows. Let $A'$ be the result of rounding $A$ to
  $c\log n$ bits per entry. Then for any $x \in \R^n$, we have that
  there exists an $s$ such that $A'x = A(x + s)$ and $||s||_1 <
  n^{2-c}$
\end{lemma}


\begin{thebibliography}{42}

\bibitem[BJKS02]{BJKS}
Z.~Bar-Yossef, T.S.~Jayram, R.~Kumar, and D.~Sivakumar.
\newblock Information theory methods in communication complexity.
\newblock In {\em Proceedings 17th Annual IEEE Conference on
  Computational Complexity}, pages 133--142, 2002.

\bibitem[CRT06]{Candes2006} Candes, Emmanuel J., Justin K. Romberg, and Terence Tao. "Stable signal recovery from incomplete and inaccurate measurements." Communications on pure and applied mathematics 59.8 (2006): 1207-1223.

\bibitem[CDD09]{Cohen2009} Cohen, Albert, Wolfgang Dahmen, and Ronald DeVore. "Compressed sensing and best 𝑘-term approximation." Journal of the American Mathematical Society 22.1 (2009): 211-231.

\bibitem[DIPW10]{Price2010} Do Ba, K., Indyk, P., Price, E., \& Woodruff, D. P. (2010, January). Lower Bounds for Sparse Recovery. In SODA (Vol. 10, pp. 1190-1197).
\end{thebibliography}

\end{document}
