\documentclass[11pt]{article}
\usepackage{amsmath,amssymb,amsthm}
\usepackage{graphicx}
\DeclareMathOperator*{\E}{\mathbb{E}}
\let\Pr\relax
\DeclareMathOperator*{\Pr}{\mathbb{P}}

\newcommand{\eps}{\epsilon}
\newcommand{\inprod}[1]{\left\langle #1 \right\rangle}
\newcommand{\R}{\mathbb{R}}

\newcommand{\handout}[5]{
  \noindent
  \begin{center}
  \framebox{
    \vbox{
      \hbox to 5.78in { {\bf CS 395T: Sublinear Algorithms } \hfill #2 }
      \vspace{4mm}
      \hbox to 5.78in { {\Large \hfill #5  \hfill} }
      \vspace{2mm}
      \hbox to 5.78in { {\em #3 \hfill #4} }
    }
  }
  \end{center}
  \vspace*{4mm}
}

\newcommand{\lecture}[4]{\handout{#1}{#2}{#3}{Scribe: #4}{Lecture #1}}

\newtheorem{theorem}{Theorem}
\newtheorem{corollary}[theorem]{Corollary}
\newtheorem{lemma}[theorem]{Lemma}
\newtheorem{observation}[theorem]{Observation}
\newtheorem{proposition}[theorem]{Proposition}
\newtheorem{definition}[theorem]{Definition}
\newtheorem{claim}[theorem]{Claim}
\newtheorem{fact}[theorem]{Fact}
\newtheorem{assumption}[theorem]{Assumption}

% 1-inch margins, from fullpage.sty by H.Partl, Version 2, Dec. 15, 1988.
\topmargin 0pt
\advance \topmargin by -\headheight
\advance \topmargin by -\headsep
\textheight 8.9in
\oddsidemargin 0pt
\evensidemargin \oddsidemargin
\marginparwidth 0.5in
\textwidth 6.5in

\parindent 0in
\parskip 1.5ex

\begin{document}

\lecture{15 --- Oct. 18, 2016}{Fall 2016}{Prof.\ Eric Price}{Ziyang Tang \& Mei Wang}

\section{Overview}

In last class, we discussed Compressed Sensing with Interactive Hard Thresholding (IHT). 
In this lecture, we continue to study IHT algorithm first and then we start to look at ``Model-Based" Compressed Sensing. 



\section{Recall for IHT Algorithm}

\begin{definition} \label{def1}
Suppose matrix $A \in \R^{m\times n}$ has $(k, \eps) RIP$. If for all $k$-sparse $x$, $\|Ax\|_2^2 = (1 \pm \eps) \|x\|_2^2$. Then this is equivalent to saying as $\|(A^\top A -I)_{S*S}\| \le \eps$ for any $S \le k$, $S \in [n]$.
\end{definition}

\begin{theorem}
If $A$ has $(3k,0.1) RIP$, given $y=Ax+e$ where $x$ is $k$-sparse, then IHT recovers $\hat{x}$ with $\| \hat{x} - x\|_2 \le \|e\|_2$.
\end{theorem}

This theorem indicates that if there is no error, you can recover it exactly. If there is some small error $\eps$, you will recover with this small norm error.

So how this theorem expected to work? 
Assume $z=A^\top y = A^\top Ax + A^\top e$.
Let $H_k(z) \in \R^n := $ closest $k$-sparse approximation for any $z \in \R^n$. 
Then $\hat{x} = H_k(z)$ has 
\begin{align*}
\| \hat{x}-x \| &\le O(\eps)  \|x\| + \|(A^\top e)_S \|.
\end{align*}
By looking at Definition~\ref{def1}, for any $\omega \in \R^{|S|}$, $\| A_S \omega\| \le \sqrt{1+\eps} \| \omega \|$, which is equal to saying that the singular values of $A_S$ are less than or equal to $\sqrt{1+\eps}$. The same property holds for $A_S^\top$. So we can get
\begin{align*}
\| \hat{x}-x \| &\le O(\eps)  \|x\| + \sqrt {1+\eps} \|e\| \\ &\le  O(\eps)  \|x\| + 2\|e\|.
\end{align*}


This argue means we can get closer, where two means we can get down to the noise level fast. 

Now we will restate the algorithm in order to repeat.
Suppose we have current guess $x^{(i)}$, then we set 
\begin{align*}
x^{(i+1)} &= H_k( A^\top (y Ax^{(i)}) + x^{(i)}) \\
& = x^{(i)} + A^\top e + A^\top A (x-x^{(i)})t.
\end{align*}
Run this equation repeatedly until you get to $\| x^{(l)} -x \| \le \|e\|$ after $log(\frac{\|x\|}{\|e\|})$ iterations.

If $A$ has $(3k,0.1)$ RIP, this holds with $O(k \log \frac{n}{k})$ Gaussian rows. Basically, this is optimal in general.


So what number should we expect of the Gaussian rows?  $\Omega (k)$ words is necessary. 
In order to match ``compression", how is the basic requirement of compression?
\begin{align*}
\underbrace{\text{(identify support)}}_{k \log(\frac{n}{k}) ~\text{bits}} + \underbrace{\text{(tell values on support)}}_{k ~\text{words}}
\end{align*}
If we don't have noise, we can do that in just $k$ words. But if we want to be robust to noise, we cannot avoid the $k \log(\frac{n}{k})$ issue. This is quite similar to the lower bound for moment estimation by Gaussian noise last week.

In some circumstances, we don't need $k \log(\frac{n}{k})$ bits to describe $k$ words, because we don't expect our vector to be an arbitrary sparse vector. We can do better with \textbf{structural sparsity}, which is also known as ``Model-Based" Compressed Sensing.



\section{``Model-Based" Compressed Sensing}
\begin{definition}
Suppose $x$ is $k$-sparse and $supp(x) \in \mathcal{F}$, where $\mathcal{F} =$ set of supports. Then $m = O(k+\log |F|)$ suffices for ``Model-Based'' Compressed Sensing.
\end{definition}
We first see some examples of $\mathcal{F}$.

\paragraph{Example 1: Block Sparsity:\\}
\quad \\
If the signal comes in blocks, that is the support of x $S$ consists of several blocks. 

\begin{figure}[htbp]
\begin{center}
%\begin{figure}
\includegraphics[width=0.6\textwidth]{blocksparsity}
%\end{figure}
\caption{Block Sparsity}
\label{default}
\end{center}
\end{figure}

Suppose the number of the block is $B$, then we can pick $n/B$ disjoint block, and we need to choose $k/B$ block out of these blocks, thus the total support number of size k should be $|\mathcal{F}| = \binom{n/B} {k/B}$. So $m$ is approximately $log \binom{n/B} {k/B} \approx \frac{k}{B} \log \binom{n}{k}$.

\clearpage

\paragraph{Example 2: Tree Sparsity\\}
\quad \\
If we want to find a subset of connected tree that has $k$ nonzeros containing root, how large is the set of Tree Sparse Supports?

Well, one way to think of a connected subset of tree is to do a tranverse of a tree, and to see how many different way can we represent a traverse over k connected vertices. If we do a traverse of all the connected support of the tree, then for each vertex at some root of a subtree, there are 4 choices of next layer of its child information, that is none of its child are in the support, one of its child is in the support and both of its child are in the support. So we need 2 bits for every vertex for a tree to represent the traverse, that is totally at most $2k$ bits. So $|\mathcal{F}|\leq 2^{2k}$, which means $O(k+log|\mathcal{F}|) = O(k)$ suffices.
\begin{figure}[htbp]
\begin{center}
%\begin{figure}
\includegraphics[width=0.48\textwidth]{tree}
\includegraphics[width=0.48\textwidth]{sparsfig}
%\end{figure}
\caption{Tree Sparsity}
\label{tree}
\end{center}
\end{figure}

Tree sparsity can also be used in sparse images. As we can see in the right part of Figure.~\ref{tree}, the sparse structure with linear operations of the figure can be modeled as a tree model. Our analysis of this tree sparsity example can be applied in image processing field.

If we assign nonnegative value on every vertex and would like to find the maximum k-sparse support on a tree, table 1 shows the complexity of this specific problem (which we will prove in homework).
\begin{table}[htp]
\caption{algorithm complexity}
\begin{center}
\begin{tabular}{|c|c|}
\hline $nk^2$ & easy DP \\
\hline $nk$ & harder DP \\
\hline $n\log n$ & $O(1)$ approximation \\
\hline
\end{tabular}
\end{center}
\label{default}
\end{table}%

\clearpage

Now we generalize the ``Model-Based" compressed sensing.
\begin{theorem}
Suppose $A$ has $(\mathcal{F},1)$ model RIP. If for any $x$ with $supp(x) \in \mathcal{F}$, we have
$$ \| Ax\|_2^2 = (1 \pm \eps) \|x\|_2^2, $$
Then Guassian Matrix has this for $M = O(k+\log |\mathcal{F}|)$ with constant $\eps$.
\end{theorem}

So how to change the algorithm to Model-Based Compressed Sensing?
$$ H_{\mathcal{F}}(z) :=\operatorname*{\hat{z}}\limits_{minimize \|\hat{z} -z\|}~\text{with}~ supp(\hat{z}) \in \mathcal{F}$$



\begin{proof}
Analysis uses RIP on $supp(x^{(i)}) \cup supp(x^{(i+1)}) \cup supp(x)$. 
So we just need RIP on $\mathcal{F} \otimes \mathcal{F} \otimes \mathcal{F}$ with size less than $\|\mathcal{F}\|^3$.
If $A$ has $(\mathcal{F}^3, 1)$ RIP, then it has $O(k+log|\mathcal{F}|)$ Gaussian rows.

We need for any $z$, $\|x-H_{\mathcal{F}}(z)\| \le \|x-z\|$.

needs an analysis that doesn't just pair up elements in $supp(x)$ and $supp(H_k(x))$.
\end{proof}

%\paragraph{Bibliography.}
%Please give real bibliographical citations for the papers that we
%mention in class. See below for how to include a bibliography section.
%If you use BibTeX, integrate the .bbl file into your .tex source. You
%should reference papers like this: ``The tug of war sketch originates
%in a paper by Alon, Matias and Szegedy \cite{AMS99}.''  In general,
%the name of the authors should appear in text at most once (for the
%first citation); further citations look like: ``Our proof follows that
%of \cite{AMS99}''.
%
%Take a look at previous lectures (TeX files are available) to see the
%details. A excellent source for bibliographical citations is
%DBLP. Just Google DBLP and an author's name.
%
%
%
%\bibliographystyle{alpha}
%
%\begin{thebibliography}{42}
%
%\bibitem[DPD14]{DPD14}
%Cormode~Graham, and Donatella~Firmani.
%\newblock A unifying framework for $l_0$-sampling algorithms.
%\newblock {\em Distributed and Parallel Databases}, 32(3):315-335, 2014.
%
%\end{thebibliography}
%


\end{document}