\documentclass[11pt]{article}
\usepackage{amsmath,amssymb,amsthm}
\usepackage{graphicx}

\DeclareMathOperator*{\E}{\mathbb{E}}
\let\Pr\relax
\DeclareMathOperator*{\Pr}{\mathbb{P}}
\DeclareMathOperator*{\var}{\mathrm{Var}}

\newcommand{\eps}{\epsilon}
\newcommand{\inprod}[1]{\left\langle #1 \right\rangle}
\newcommand{\R}{\mathbb{R}}
\newcommand{\Z}{\mathbb{Z}}
\renewcommand{\vec}[1]{\boldsymbol{\mathrm{#1}}}
\newcommand{\HH}{\textrm{HH}_\alpha}
\newcommand{\pleq}{\mathrel{+}=}

\newcommand{\handout}[5]{
  \noindent
  \begin{center}
  \framebox{
    \vbox{
      \hbox to 5.78in { {\bf CS388R: Randomized Algorithms } \hfill #2 }
      \vspace{4mm}
      \hbox to 5.78in { {\Large \hfill #5  \hfill} }
      \vspace{2mm}
      \hbox to 5.78in { {\em #3 \hfill #4} }
    }
  }
  \end{center}
  \vspace*{4mm}
}

\newcommand{\lecture}[4]{\handout{#1}{#2}{#3}{Scribe: #4}{Lecture #1}}

\newtheorem{theorem}{Theorem}
\newtheorem{corollary}[theorem]{Corollary}
\newtheorem{lemma}[theorem]{Lemma}
\newtheorem{observation}[theorem]{Observation}
\newtheorem{proposition}[theorem]{Proposition}
\newtheorem{definition}[theorem]{Definition}
\newtheorem{claim}[theorem]{Claim}
\newtheorem{fact}[theorem]{Fact}
\newtheorem{assumption}[theorem]{Assumption}

% 1-inch margins, from fullpage.sty by H.Partl, Version 2, Dec. 15, 1988.
\topmargin 0pt
\advance \topmargin by -\headheight
\advance \topmargin by -\headsep
\textheight 8.9in
\oddsidemargin 0pt
\evensidemargin \oddsidemargin
\marginparwidth 0.5in
\textwidth 6.5in

\parindent 0in
\parskip 1.5ex

\begin{document}

\lecture{18 --- Nov 09, 2015}{Fall 2015}{Prof.\ Eric Price}{Jin Zhang, Rishab Goyal}

\section{Overview}

In the previous lecture, we discussed 
\begin{enumerate}
\item The minimum sample size to estimate fraction $p$ of an unknown set, conforming certain property, within absolute relative error $\eps$ and at most $\delta$ failure probability.
\item How to find median of a list of elements with time complexity $\frac{3}{2}n + o(n)$.
\item (Briefly) the streaming algorithm to find $\alpha$-heavy hitters ($\HH$) after seeing $v_1, v_2, \ldots, v_m$ items, deterministically.
\end{enumerate}


In this lecture, we revisit the count-min sketch problem, and analyze it in detail. First, we recall the \emph{turnstile} model, and introduce some notation that we will follow throughout the scribe.

\section{Introduction}
Streaming algorithms are (informally) defined as algorithms that work on (massive) streams of data given only limited space and computational resources. The data stream model we will be interested in is called \emph{turnstile} model. In turnstile model, each update is of the form $(v_i, t_i)$, where $v_i$ is a data point/ item, and $t_i \in \Z$ is a count. Throughout the scribe, we will denote by $x_{u}$, the number of times item $u$ appeared in the stream. Concretely, after each update $(v_i, t_i)$, $x_{v_i} \pleq t_i$. Also, we define $\vec{x} = (x_{u_1}, \ldots, x_{u_n})$, $n = ||\vec{x}||_0$ and $m = ||\vec{x}||_1$, where $u_i$'s are all distinct items.

In the standard turnstile model, $x_{v_i}$ could be negative, however here we consider \emph{strict} turnstile model, where $x_{v_i}$ is non-negative for all items $v_i$ at all time steps. We would like to find the set of $\alpha$-heavy hitters ($\HH$), where $\HH := \{ u\ |\ x_u \geq \alpha m\}$, where $m, n$ are same as defined before. Instead of solving perfectly, we solve an approximate version of the heavy hitters problem in the strict turnstile model. The $(\eps, \delta)-\HH$ is defined as finding a set $S$ subject to the constraint that $S \supseteq \HH$ and $\forall u \in S$, $x_u \geq (1 - \eps) \alpha m$, with failure probability at most $\delta$. We could even find a set $S$ such that $\forall u \in S$, $x_u \in [(1 - \eps) \alpha m, (1 + \eps) \alpha m]$ using the same idea.

In the next section, we describe count-min sketch, and show how to use it to solve $(\eps, \delta)-\HH$.


\section{Count-min sketch}
Count-min sketch could be considered as a counting Bloom filter using pairwise independent hash functions. In count-min sketch, we first make a sketch $Y \in \mathbb{Z}^{r \times c}$, such that $r = O\left(\log \dfrac{1}{\delta}\right)$, $c = O\left(\dfrac{1}{\alpha}\right)$. For each row $i$, we use a different pairwise independent hash function $h_i: U \rightarrow [c]$, and each element in $y$ satisfies:

\begin{equation*}
y_{i,u} = \sum\limits_{v|h_i(v) = u} x_v
\end{equation*}

Every update $(v, t)$ modifies the sketch $Y$ as follows (Figure 1)
\begin{equation*}
\forall i \in [r],\quad y_{i, h_i(v)} \pleq t.
\end{equation*}

\begin{figure}[h]
\begin{center}
\includegraphics[width=180mm,trim=1cm 8cm 3cm 1.2cm]{cmsketch.jpg}
\end{center}
\caption{Updating count-min sketch}
\label{tab:f1}
\end{figure}

Given the sketch $Y$, we define $\tilde{x_v}^{(i)} = y_{i, h_i(v)}$ as the estimate of $x_v$ from $i^{th}$ row. It is straightforward to see that we always overestimate the value because $x_v \ge 0\ \forall v$ under the strict turnstile model. Therefore, to get a better estimate for $x_v$, we compute $\tilde{x_v}$ as

\begin{equation*}
\tilde{x_v} = \min_i \tilde{x_v}^{(i)}.
\end{equation*}

\begin{claim}
(Informal) $Z_i = \tilde{x_v}^{(i)} - x_v$ is small $\left(\leq 2 \dfrac{||\vec{x}||_1}{c}\right)$ with probability $\geq \dfrac{1}{2}$.
\end{claim}

\begin{proof} In order to prove the above claim, we show an upper bound on $\E[Z]$, and then apply markov's inequality.

\begin{align}
\E[\tilde{x_v}^{(i)} - x_v] &= \E\left[\sum_{v' \ne v} x_{v'} \cdot I_{h_i(v') = h_i(v)}\right] \\
&= \sum_{v' \ne v} x_{v'} \cdot \E[I_{h_i(v') = h_i(v)}] \\
&= \sum_{v' \ne v} x_{v'} \cdot \dfrac{1}{c} \\
&\le \dfrac{\|\vec{x}\|_1}{c}
\end{align}

$I_{h_i(v') = h_i(v)}$ denotes the indicator function which is $1$ \emph{iff} $h_i(v')$ and $h_i(v)$ collide. Equation (2) to (3) follow from pairwise independence of $h_i$.

\end{proof}


Now we use markov's inequality to obtain,

\begin{align*}
\Pr\left(\E[\tilde{x_v}^{(i)} - x_v] \le \frac{2\|\vec{x}\|_1}{c}\right) &\le \frac{1}{2} \\
\Rightarrow \Pr\left(\E[\tilde{x_v} - x_v] \le \frac{2\|\vec{x}\|_1}{c}\right) &\le 1 - \frac{1}{2^r} = 1 - \delta.
\end{align*}

Let $c = \dfrac{2}{\epsilon \alpha}$, we get 

\begin{align*}
\Pr\left(\E[\tilde{x_v} - x_v] \le \frac{2\|\vec{x}\|_1}{c}\right) &= \Pr\left(\E[\tilde{x_v} - x_v] \le \epsilon \alpha \|\vec{x}\|_1\right) \\
&= \Pr(\E[\tilde{x_v} - x_v] \le \epsilon \alpha m) \\
&\le 1 - \delta
\end{align*}


Now to find the $(\eps, \delta)$-heavy hitters (find $S \supseteq \HH$), we look at all $v \in U$, and take $S := \{v\ |\ \tilde{x_v} \geq \alpha m\}$. First, observe that if $x_v \geq \alpha m$, then $\tilde{x_v} \geq x_v \geq \alpha m$, therefore $v \in S$. Second, if $x_v < (1 - \eps) \alpha m$, then $v$ does not satisfy the criterion of belonging to $(\eps, \delta)$-$\HH$, and we know from above inequalities that the probability $\tilde{x_v} \geq (1 - \eps) \alpha m$ given $x_v < (1 - \eps) \alpha m$ is at most $\delta = 1/ 2^r$. Therefore, the probability of any item being misclassified as $\eps-\HH$ is at most $1 / 2^r$, so using union bound, we could write that

\begin{equation*}
\E[\text{No. of incorrect classifications}] \le |U| \cdot \delta
\end{equation*}

Therefore, if we replace $r$ from $\log \dfrac{1}{\delta}$ to $\log \dfrac{|U|}{\delta}$, we get $\delta' =  \frac{\delta}{|U|}$, and $E[\text{No. of incorrect classifications}] \le |U|\delta' = \delta$. Therefore, using markov inequality
\begin{equation*}
\Pr(\E[\text{No. of incorrect classifications}] \ge 1) \le \delta
\end{equation*}


So now we get the set $S$ as $(\eps, \delta)-\HH$ heavy hitter approximation with probability $1 - \delta$, and the time complexity is $O\left(\dfrac{1}{\epsilon \alpha} \log \dfrac{|U|}{\delta}\right)$.


\subsection{Improvement of error bound}
Previous analysis make no assumption on the data stream, however, we observe that many kinds of real data has skewed distribution, such as power distribution, where the count of $i^{th}$ item $x_i = C \cdot i^{-s}$, where $C$ is a constant, and $s > 1$. When we estimate the count of the item, and when it collides with the first k large items, our error estimation will be much better than previously described.\\

\begin{claim}
(Informal) We can achieve error approximately $C \dfrac{\alpha^{s-1}}{s-1}$.
\end{claim}

Let $H \subseteq U$ denote the top $c / 4$ items of $\vec{x}$. Let $\vec{x}_{-k}$ be the set of items except for the first $k$ heavy items. The collision probability is:

\begin{equation}
\forall v \in U,\quad \Pr(h_i(v) \in \{h_i(v')\ |\ v' \in H, v' \ne v\}) \le \frac{|H|}{c} = \frac{1}{4}.
\end{equation}

Let error $Z = \sum_{v' \ne v, h_i(v') = h_i(v), v \notin H} x_v$. We have
\begin{equation*}
\E[Z] = \frac{1}{c} \sum_{v' \ne v, v \notin H} x_v = \frac{\|\vec{x}_{-\frac{c}{4}}\|_1}{c}
\end{equation*}

\begin{equation}
\Pr\left(Z \ge 4\frac{\|\vec{x}_{-\frac{c}{4}}\|_1}{c}\right) \le \frac{1}{4}
\end{equation}

Also note
\begin{equation}
\|\vec{x}_{-k}\|_1 = C \sum_{k+1}^{\infty} \frac{1}{i^s}  \le C \int_{k}^{\infty} \frac{1}{i^s} di = C \frac{1}{s-1} \frac{1}{k^{s-1}}.
\end{equation}

So the probability neither (5) or (6) happen is $\ge \frac{1}{2}$, with error $O\left(\dfrac{C}{c(s - 1)} \left(\dfrac{4}{c}\right)^{s-1}\right)$, or $O\left(C \dfrac{\alpha^{s-1}}{s-1}\right)$ because $c = O\left(\dfrac{1}{\alpha}\right)$.


\subsection{Improvement of time complexity}
Now we have shown that with count-min sketch, we can compute $\eps$ heavy hitter $S \supseteq \{v\ |\ x_v \ge \alpha \|\vec{x}\|_1 \}$, $|S| \le \dfrac{2}{\alpha}$ with $O\left(\dfrac{1}{\alpha}\log |U|\right)$ space. Next, we try to improve the time complexity.\\

Observe that if $v$ is heavy ($x_v \ge \alpha \|x\|_1$), for all $S \supseteq \{v\}$, we have $\sum_{v \in S} x_v \ge \alpha \|x\|_1$, or these sets are heavy as well.\\

We solve the heavy hitter problem using a divide and conquer procedure, using $\log |U|$ sketches. The first sketch will count the two subsets separated based on the difference of the first bit of each item. The second sketch count subsets on the difference of the first two bits within the four subsets. For sketch $t$, we have
\begin{equation*}
y_{t, i, u} = \sum_{v:h_{t,i}} (\text{first } t \text{ bits of v} == u)
\end{equation*}

To get all the heavy hitters, we perform a binary search on these subsets from sketch 1 to $\log |U|$, as described before, at each level, if the subset $S$ contains the heave hitter, the subset itself is heavy. We trace the heavy subset to the last sketch and output all the heavy single items. The time complexity in this case is $O\left(\dfrac{1}{\alpha} \log^2|U|\right)$.


\begin{thebibliography}{42}

 \bibitem[MR]{MR}
 Rajeev~Motwani, Prabhakar~Raghavan
 \newblock Randomized Algorithms.
 \newblock {\em Cambridge University Press}, 0-521-47465-5, 1995. %pub - isbn - year

 \bibitem[CM]{CM}
 G. Cormode and S. Muthukrishnan. 
 \newblock What’s hot and what’s not: Tracking most frequent items dynamically.
 \newblock {\em In Proceedings of ACM Principles of Database Systems}, pages 296-306, 2003.

 \bibitem[CM]{CM2}
 G. Cormode and S. Muthukrishnan.
 \newblock An improved data stream summary: The count-min sketch and its applications.
 \newblock {\em Journal of Algorithms}, 55(1):58–75, 2005.


\end{thebibliography}

\end{document}