Commit defbcae6 authored by Parth Mittal's avatar Parth Mittal
Browse files

wrote count-min and the AMS estimator for distinct

parent 9744cd1b
......@@ -43,7 +43,7 @@ us with a small set $C$ containing $F_k$, and hence lets us solve the frequent
elements problem efficiently.
\algo{FrequencyEstimate} \algalias{Misra/Gries Algorithm}
\algin the data stream $\alpha$, the target for the estimator $k$
\algin the data stream $\alpha$, the target for the estimator $k$.
\:\em{Init}: $A \= \emptyset$. \cmt{an empty map}
\::If $x \in$ keys($A$), $A[x] \= A[x] + 1$.
......@@ -93,12 +93,138 @@ $\vert C \vert = \vert$keys($A$)$\vert \leq k - 1$, and a key-value pair can
be stored in $\O(\log n + \log m)$ bits.
\subsection{The Count-Min sketch}
\subsection{The Count-Min Sketch}
We will now look at a randomized streaming algorithm that solves the
frequency estimation problem. While this algorithm can fail with some
probability, it has the advantage that the output on two different streams
can be easily combined.
\algo{FrequencyEstimate} \algalias{Count-Min Sketch}
\algin the data stream $\alpha$, the accuracy $\varepsilon$,
the error parameter $\delta$.
$C[1\ldots t][1\ldots k] \= 0$, where $k \= \lceil 2 / \varepsilon \rceil$
and $t \= \lceil \log(1 / \delta) \rceil$.
\:: Choose $t$ independent hash functions $h_1, \ldots h_t : [n] \to [k]$, each
from a 2-independent family.
\::For $i \in [t]$: $C[i][h_i(x)] \= C[i][h_i(x)] + 1$.
\algout Report $\hat{f}_a = \min_{i \in t} C[i][h_i(a)]$.
We will now look at a randomized streaming algorithm that performs the same task
Note that the algorithm needs $\O(tk \log m)$ bits to store the table $C$, and
$\O(t \log n)$ bits to store the hash functions $h_1, \ldots h_t$, and hence
uses $\O(1/\varepsilon \cdot \log (1 / \delta) \cdot \log m
+ \log (1 / \delta)\cdot \log n)$ bits. It remains to show that it computes
a good estimate.
$f_a \leq \hat{f}_a \leq f_a + \varepsilon m$ with probability $\delta$.
Clearly $\hat{f}_a \geq f_a$ for all $a \in [n]$; we will show that
$\hat{f}_a \leq f_a + \varepsilon m$ with probability at least $\delta$.
For a fixed element $a$, define the random variable
$$X_i := C[i][h_i(a)] - f_a$$
For $j \in [n] \setminus \{ a \}$, define the
indicator variable $Y_{i, j} := [ h_i(j) = h_i(a) ]$. Then we can see that
$$X_i = \sum_{j \neq a} f_j\cdot Y_{i, j}$$
Note that $\E[Y_{i, j}] = 1/k$ since each $h_i$ is from a 2-independent family,
and hence by linearity of expectation:
$$\E[X_i] = {\vert\vert f \vert\vert_1 - f_a \over k} =
{\vert\vert f_{-a} \vert\vert_1 \over k}$$
And by applying Markov's inequality we obtain a bound on the error of a single
$$ \Pr[X_i > \varepsilon \cdot m ] \geq
\Pr[ X_i > \varepsilon \cdot \vert\vert f_{-a} \vert\vert_1 ] \leq
{1 \over k\varepsilon} \leq 1/2$$
Finally, since we have $t$ independent counters, the probability that they
are all wrong is:
$$ \Pr\left[\bigcap_i X_i > \varepsilon \cdot m \right] \leq 1/2^t \leq \delta $$
\section{Counting Distinct Elements}
We continue working with a stream $\alpha[1 \ldots m]$ of integers from $[n]$,
and define $f_a$ (the frequency of $a$) as before. Let
$d = \vert \{ j : f_j > 0 \} \vert$. Then the distinct elements problem is
to estimate $d$.
\subsection{The AMS Algorithm}
Define ${\tt tz}(x) := \max\{ i \mid 2^i $~divides~$ x \}$
(i.e. the number of trailing zeroes in the base-2 representation of $x$).
\algo{DistinctElements} \algalias{AMS}
\algin the data stream $\alpha$, the accuracy $\varepsilon$,
the error parameter $\delta$.
\:\em{Init}: Choose a random hash function $h : [n] \to [n]$ from a 2-independent
\:: $z \= 0$.
\::If ${\tt tz}(h(x)) > z$: $z \= {\tt tz}(h(x))$.
\algout $\hat{d} \= 2^{z + 1/2}$
The AMS algorithm is a $(3, \delta)$-estimator for some constant
For $j \in [n]$, $r \geq 0$, let $X_{r, j} := [ {\tt tz}(h(j)) \geq r ]$, the
indicator that is true if $h(j)$ has at least $r$ trailing $0$s.
Now define $$ Y_r = \sum_{j : f_j > 0} X_{r, j} $$
How is our estimate related to $Y_r$? If the algorithm outputs
$\hat{d} \geq 2^{a + 1/2}$, then we know that $Y_a > 0$. Similarly, if the
output is smaller than $2^{a + 1/2}$, then we know that $Y_a = 0$. We will now
bound the probabilities of these events.
For any $j \in [n]$, $h(j)$ is uniformly distributed over $[n]$ (since $h$
is $2$-independent). Hence $\E[X_{r, j}] = 1 / 2^r$. By linearity of
expectation, $\E[Y_{r}] = d / 2^r$.
We will also use the variance of these variables -- note that
$${\rm Var}[X_{r, j}] \leq \E[X_{r, j}^2] = \E[X_{r, j}] = 1/2^r$$
And because $h$ is $2$-independent, the variables $X_{r, j}$ and $X_{r, j'}$
are independent for $j \neq j'$, and hence:
$${{\rm Var}}[Y_{r}] = \sum_{j : f_j > 0} {\rm Var}[X_{r, j}] \leq d / 2^r $$
Now, let $a$ be the smallest integer such that $2^{a + 1/2} \geq 3d$. Then we
$$ \Pr[\hat{d} \geq 3d] = \Pr[Y_a > 0] = \Pr[Y_a \geq 1] $$
Using Markov's inequality we get:
$$ \Pr[\hat{d} \geq 3d] \leq \E[Y_a] = {d \over 2^a} \leq {\sqrt{2} \over 3} $$
For the other side, let $b$ be the smallest integer so that
$2^{b + 1/2} \leq d/3$. Then we have:
$$ \Pr[\hat{d} \leq d / 3] = \Pr[ Y_{b + 1} = 0] \leq
\Pr[ \vert Y_{b + 1} - \E[Y_{b + 1}] \vert \geq d / 2^{b + 1} ]$$
Using Chebyshev's inequality, we get:
$$ \Pr[\hat{d} < d / 3] \leq {{\rm Var}[Y_b] \over (d / 2^{b + 1})^2} \leq
{2^{b + 1} \over d} \leq {\sqrt{2} \over 3}$$
The previous algorithm is not particularly satisfying -- by our analysis it
can make an error around $94\%$ of the time (taking the union of the two bad
events). However we can improve the success probability easily; we run $t$
independent estimators simultaneously, and print the median of their outputs.
By a standard use of Chernoff Bounds one can show that the probability that
the median is more than $3d$ is at most $2^{-\Theta(t)}$ (and similarly also
the probability that it is less than $d / 3$).
Hence it is enough to run $\O(\log (1/ \delta))$ copies of the AMS estimator
to get a $(3, \delta)$ estimator for any $\delta > 0$. Finally, we note that
the space used by a single estimator is $\O(\log n)$ since we can store $h$ in
$\O(\log n)$ bits, and $z$ in $\O(\log \log n)$ bits, and hence a $(3, \delta)$
estimator uses $\O(\log (1/\delta) \cdot \log n)$ bits.
Supports Markdown
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment