\documentclass[11pt]{article}
\usepackage{amsmath,amssymb,amsthm,mathtools}

\DeclareMathOperator*{\E}{\mathbb{E}}
\let\Pr\relax
\DeclareMathOperator*{\Pr}{\mathbb{P}}
\DeclareMathOperator*{\argmin}{arg\,min}

\newcommand{\eps}{\epsilon}
\newcommand{\inprod}[1]{\left\langle #1 \right\rangle}
\newcommand{\R}{\mathbb{R}}
\newcommand{\poly}{{\rm poly}}

\newcommand{\handout}[5]{
  \noindent
  \begin{center}
  \framebox{
    \vbox{
      \hbox to 5.78in { {\bf CS 395T: Sublinear Algorithms } \hfill #2 }
      \vspace{4mm}
      \hbox to 5.78in { {\Large \hfill #5  \hfill} }
      \vspace{2mm}
      \hbox to 5.78in { {\em #3 \hfill #4} }
    }
  }
  \end{center}
  \vspace*{4mm}
}

\newcommand{\lecture}[4]{\handout{#1}{#2}{#3}{Scribe: #4}{Lecture #1}}

\newtheorem{thm}{Theorem}[section]
\newtheorem{claim}[thm]{Claim}
\newtheorem{lemma}[thm]{Lemma}
\newtheorem{define}[thm]{Definition}
\newtheorem{cor}[thm]{Corollary}
\newtheorem{obs}[thm]{Observation}
\newtheorem{remark}[thm]{Remark}
\newtheorem{example}[thm]{Example}
\newtheorem{construction}[thm]{Construction}
\newtheorem{conjecture}{Conjecture}
\newtheorem{THM}{Theorem}
\newtheorem{COR}[THM]{Corollary}
\newtheorem{question}{Question}
\newtheorem{fact}[thm]{Fact}
\newtheorem{prop}[thm]{Proposition}
% 1-inch margins, from fullpage.sty by H.Partl, Version 2, Dec. 15, 1988.
\topmargin 0pt
\advance \topmargin by -\headheight
\advance \topmargin by -\headsep
\textheight 8.9in
\oddsidemargin 0pt
\evensidemargin \oddsidemargin
\marginparwidth 0.5in
\textwidth 6.5in

\parindent 0in
\parskip 1.5ex

\begin{document}

\lecture{1 --- Aug, 30, 2016}{Fall 2016}{Prof.\ Eric Price}{John Kallaugher, Akshay Kamath }

\section{Overview}

In this lecture, we will overview the three areas of computer science which we will cover in this course. 

\begin{enumerate}
\item Property Testing: There is a large amount of data which we can `query' locally and we wish to test whether the data satisfies some property using $o(n)$ queries.
\item Streaming Algorithms: The data arrives in a stream and we wish to evaluate some statistic of the data while using $o(n)$ space.
\item Compressed Sensing: We wish to make a $o(n)$ `measurements' of the data and then compute functions of the data.

\end{enumerate}

\section{ Property Testing}
\subsection{Introduction}
Some well known property testing problems are:
\begin{enumerate}
\item Bipartiteness Testing: Is the graph bipartite?
\item Connectedness Testing: Is the graph connected?
\item Uniformity Testing: Is the given distribution uniform?
\end{enumerate}

One might argue that the problems stated above cannot be solved by making sublinear queries. For example, there may be a bipartite graph with just one edge within a bipartition. 

So, we relax the problem slightly and phrase it as follows:\\
{\bf Distinguish between}
\begin{enumerate}
\item $X$ has property $P$: Accept with high probabilty.
\item $X$ is a $\epsilon$-far from satisfying $P$: Reject with high probability.
\end{enumerate}

\subsection{Uniformity Testing}

The uniformity testing problem may be phrased as follows:\\
{\bf Distinguish between}
\begin{enumerate}
\item $X$ is the uniform distribution on $[n]$: Accept with high probability 
\item $X$ is $\epsilon$-far in TV distance from uniform on $[n]$: Reject with high probability.
\end{enumerate}

In this case, the distance metric used is Total Variation distance:
$\| X - \mathcal{U}_n\|_{\text{TV}} = \frac{1}{2}\sum\limits_{i=0}^{n} |X_i - \frac{1}{n}|$\\

\textbf{Toy Problem:}
{\bf Distinguish between}
\begin{enumerate}
\item $X$ is the uniform distribution on $[n]$: Accept with high probability 
\item $X$ is uniform over $S\subseteq [n]$, $|S|=\frac{n}{2}$: Reject with high probability.
\end{enumerate}

In order to distinguish between these two cases, we may sample until we see more than $n/2$ distinct elements. 

If our distribution is from the first case, observe that if we have sampled less than $n/2$ elements, the probability of sampling a new element is more than $\frac{1}{2}$. So, we will need $\approx n$ draws until we see more than $n/2$ elements. So, we use $O(n)$ samples. 

If our distribution lies in the second case, we never see more than $n/2$ elements and always reject.\\
\qed

In this course, we will deal with more sophisticated methods that will give us better results.

If we wish to accept/reject with constant probability, we know that $O(\frac{\sqrt{n}}{\epsilon^2})$ samples suffice for Uniformity Testing problem. 

If we wish to accept/reject with $\delta$ probability, we knew that $O(\frac{\sqrt{n}}{\epsilon^2}\log(\frac{1}{\delta}))$. A  recent result improves this to $O(\frac{\sqrt{n\log(\frac{1}{\delta})}}{\epsilon^2} + \frac{\log(\frac{1}{\delta})}{\epsilon^2})$ 

\section{Streaming Algorithms}
\subsection{Introduction}
We assume we have a ``stream'' of data passing by, that is too large to keep in main storage.

\begin{itemize}
    \item Orders being processed.
    \item Connections through a router.
    \item Scanning off disk into RAM.
\end{itemize}

We can divide these problems into two cases.

\begin{description}
    \item[Single Pass] We only get to see the data once ever. (e.g.\ router connections)
    \item[Multiple Pass] Accessing the data stream is expensive, but we can do it multiple times, so we can make a small (usually constant or logarithmic) number of passes over the stream.
\end{description}

Our goal is to compute some function of the data, while using sublinear space, often $O(n^\epsilon)$ or $O(\log^c n)$\footnote{This is typically referred to as ``polylog'' space}.

\subsection{Distinct Elements}
In the distinct elements problem, we receive a stream of elements from a ``universe'' $\lbrack U \rbrack$. The stream contains $n$ elements, with $k$ distinct elements.

The na\"ive algorithm takes $O(k \log U)$ bits, by storing every unique element seen so far. If we want to compute $k$ with total accuracy, even with only, say, $9/10$ success probability, this many bits is necessary. (see problem set 1)

Instead, we will attempt to find an $(\epsilon,\delta)$ approximation---with probability $1 - \delta$, our estimate should be between  $(1-\epsilon)k$ and $(1 + \epsilon)k$. A series of algorithms have steadily improved the space bounds for this problem.

\begin{description}
    \item[Probabilistic Counting] $O(\frac{1}{\epsilon^2} \log U)$ space.
    \item[LogLog] $O(\frac{1}{\epsilon^2}\log\log U)$ space.
    \item[HyperLogLog] As above, but with better constants. Widely used in practice.
    \item[Theoretical Result] $O(\frac{1}{\epsilon^2} + \log U)$. Not a practical algorithm.
\end{description}

\section{Compressed Sensing}
\subsection{Introduction}
We want to estimate some vector $X \in \mathbb{R}^n$.

\begin{itemize}
\item Image
\item Audio
\item Geological Strata (seismic sensing)
\end{itemize}

But we can't observe the data directly. Instead, we have some ability to take ``measurements'' of it.

\begin{itemize}
\item In an MRI scan, we get samples from the fourier transform of brain waves.
\item In a single-pixel camaera, there is a single photosensitive element with a mask in front of it, as opposed to the many distinct photosensitive elements in a standard camera.
\end{itemize}

How can we recover $X$ from these measurements? Consider the related problem of \emph{compression}. There, we use the structure of a signal to compress the signal into a smaller representation, then recover the signal from that representiation. (approximately, in the case of lossy compression methods such as mp3s)

Perhaps with the right measurements we can recover $X$ without ever observing the whole thing?

\subsection{General Formulation}
We need to define what we mean by ``measurements''. You may choose $A \in \mathbb{R}^{m \times n}$. You may then sense $AX \in \mathbb{R}^m$. (we can think of $AX$ as being $m$ measurements of $X$)

But what do we mean by ``structure''? Here there are many options. One is sparsity.

\begin{description}
    \item[Exact Sparsity] $X$ has at most $k$ non-zero values.
    \item[Approximate Sparsity] $\min ||X - X_k||$ is ``small'', with the minimum taken over $k$-sparse vectors $X_k$.
\end{description}

So in this case, given $Y=AX$ and $A$, we will attempt to find an estimate $\hat{X}$ of $X$ with error bounded by the lack of sparsity in $X$, i.e.

\[
||\hat{X} - X|| \leq C \min_{\text{$k$-sparse $X_k$}} ||X-X_k||
\]

\subsection{Example}
If $A \sim \mathcal{N}(0,I_{m \times n})$, with $m \geq O(k \log \frac{n}{k})$ then, with high probability, $A$ will satisfy the restricted isometry property. (the RIP)

If $A$ satisfies the RIP then

\[
\forall X,\ \hat{X} \coloneqq \argmin_{||A\hat{X} - Y|| \leq \epsilon} ||\hat{X}||
\]

satisfies $||\hat{X} - X||_2 \leq C||X - X_k||_2$ with high probability.

\end{document}
