Category Archive

You are currently browsing the category archive for the 'theory' category.

Max Cut and the Smallest Eigenvalue

June 13, 2008 in math, theory | Tags: Expanders, Max Cut, Spectral partitioning | | 3 comments

In the Max Cut problem, we are given an undirected graph $G=(V,E)$ and we want to find a partition $(L,\bar L)$ of the set of vertices such that as many edges as possible have one endpoint in $L$ and one endpoint in $\bar L$ , and are hence cut by the partition.

It is easy, as recognized since the 1970s, to find a partition that cuts half of the edges and that, thus, is at least half as good as an optimal solution. No approximation better than 1/2 was known for this problem, until Goemans and Williamson famously proved that one could achieve a .878… approximation using semidefinite programming (SDP).

No other approximation algorithm achieving an approximation asymptotically better than 1/2 is known, and it seems that a fundamental difficulty is the following. Suppose we prove that a certain algorithm achieves approximation $> 51\%$ . Then, given a graph in which the optimum is, say, $< 50.4 \%$ , the algorithm and its analysis must provide a certificate that the optimum cut in the given graph is $< 50.4/51 < 99\%$ , and there is no general technique to prove upper bounds to the Max Cut optimum of a general graph other than Semidefinite Programming. (And see here and here for negative results showing that large classes of linear programming relaxations are unable to give such certificates.)

Spectral techniques can prove upper bounds to Max Cut in certain cases (and can be seen as special cases of the upper bounds provided by the Goemans-Williamson relaxation).

In the simplified case in which $G=(V,E)$ is a $d$ -regular graph, let $A$ be the adjacency matrix of $G$ and $\lambda_1 \geq \lambda_2 \geq \cdots \lambda_n$ be the eigenvalues of $A$ ; then it is easy to show that

$(1) \ \ \ \displaystyle maxcut(G) \leq \frac 12 + \frac 12 \cdot \frac {|\lambda_n|}{d}$

where $maxcut(G)$ is the fraction of edges cut by an optimal solution. Unfortunately (1) does not quite have an approximate converse: there are graphs where $|\lambda_n|=d$ but $maxcut(G) = \frac 12 + o_d(1)$ .

The following fact, however, is always true and well known:

$|\lambda_n|=d$ if and only if $G$ contains a bipartite connected component.

Is there an “approximate” version of the above statement characterizing the cases in which $d-|\lambda_n|$ is small? Surprisingly, as far as I know the question had not been considered before.

For comparison, the starting point of the theory of edge expansion is the related fact

$\lambda_2=d$ if and only if $G$ is disconnected.

Which can be rephrased as:

$\lambda_2=d$ if and only if there is a non-empty $S\subseteq V$ , $|S| \leq |V|/2$ such that $edges(S,V-S)=0$ .

Cheeger’s inequality characterizes the case in which $d-\lambda_2$ is small:

If there is a non-empty $S\subseteq V$ , $|S| \leq |V|/2$ such that $edges(S,V-S) \leq \epsilon \cdot d \cdot |S|$ , then $\lambda_2 \geq d\cdot (1-2\epsilon)$ ;
If $\lambda_2 \geq d\cdot (1-\epsilon)$ then there is a non-empty $S\subseteq V$ , $|S| \leq |V|/2$ such that $edges(S,V-S) \leq \sqrt{2 \epsilon} \cdot d \cdot |S|$ .

For a subset $S\subseteq V$ , and a bipartition $L,R=S-L$ of $S$ , we say that an edge $(i,j)$ fails [to be cut by] the bipartition if $(i,j)$ is incident on $S$ but it is not the case that one endpoint is in $L$ and one endpoint is in $R$ . (This means that either both endpoints are in $L$ , or both endpoints are in $R$ , or one endpoint is in $S$ and one endpoint is not in $S$ .) Then we can express the well-known fact about $\lambda_n$ as

$|\lambda_n|=d$ if and only if there is $S\subseteq V$ and a bipartition of $S$ with zero failed edges.

In this new paper I prove the following approximate version

If there is a non-empty $S\subseteq V$ , and a bipartition of $S$ with at most $\epsilon \cdot d \cdot |S|$ failed edges, then $|\lambda_n| \geq d\cdot (1-4\epsilon)$ ;
If $|\lambda_n| \geq d\cdot (1-\epsilon)$ , then there is a non-empty $S\subseteq V$ , and a partition of $S$ with at most $\sqrt{2 \epsilon} \cdot d \cdot |S|$ failed edges.

The following notation makes the similarity with Cheeger’s inequality clearer. Define the edge expansion of a graph $G$ as

$\displaystyle h(G) = \min_{S\subseteq V. \ |S| \leq \frac {|V|}{2} } \frac {edges(S,V-S)} {d|S|}$

Let us define the bipartiteness ratio of $G$ as

$\displaystyle \beta(G) = \min_{S, \ L \subseteq S, \ R=S-L} \frac{edges(L) + edges(R) + edges(S,V-S)}{d|S|}$

that is, as the minimum ratio between failed edges of a partition of a set $S$ over $d|S|$ .

Then Cheeger’s inequality gives

$\displaystyle \frac 12 \cdot \frac{d-\lambda_2}{d} \leq h(G) \leq \sqrt{2 \cdot \frac{d-\lambda_2}{d} }$

and our results give

$\displaystyle \frac 14 \cdot \frac{d-|\lambda_n|}{d} \leq \beta(G) \leq \sqrt{2 \cdot \frac{d-|\lambda_n|}{d}}$

This translates into an efficient algorithm that, given a graph $G$ such that $maxcut(G) \geq 1-\epsilon$ , finds a set $S$ and a bipartition of $S$ such that at least a $1- 4\sqrt{\epsilon}$ fraction of the edges incident on $S$ are cut by the bipartition. Removing the vertices in $S$ and continuing recursively on the residual graph yields a .50769… approximation algorithm for Max Cut. (The algorithm stops making recursive calls, and uses a random partition, when the partition of $S$ found by the algorithm has too many failed edges.)

The paper is entirely a by-product of the ongoing series of posts on edge expansion: the question of relations between spectral techniques to max cut was asked by a commenter and the probabilistic view of the proof of Cheeger’s inequality that I wrote up in this post was very helpful in understanding the gap between $\lambda_n$ and $-d$ .

Dense Subsets of Pseudorandom Sets: The Paper(s)

June 5, 2008 in math, theory | Tags: Additive Combinatorics, Ben Green, Pseudorandomness, Tamar Ziegler, Terence Tao, Tim Gowers | | 1 comment

Green, Tao and Ziegler, in their works on patterns in the primes, prove a general result of the following form: if $X$ is a set, $R$ is a, possibly very sparse, “pseudorandom” susbset of $X$ , and $D$ is a dense subset of $R$ , then $D$ may be “modeled” by a large set $M$ which has the same density in $X$ as the density of $D$ in $R$ .

They use this result with $X$ being the integers in a large interval $\{ 1,\ldots, N\}$ , $R$ being the “almost-primes” in $X$ (integers with no small factor), and $D$ being the primes in $X$ . Since the almost-primes can be proved to be “pseudorandom” in a fairly strong sense, and since the density of the primes in the almost-primes is at least an absolute constant, it follows that the primes are “indistinguishable” from a large set $M$ containing a constant fraction of all integers. Since such large sets are known to contain arbitrarily long arithmetic progressions, as proved by Szemeredi, Green and Tao are able to prove that the primes too must contain arbitrarily long arithmetic progressions. Such large sets are also known to contain arbitrarily long “polynomial progressions,” as proved by Bergelson and Leibman, and this allows Tao and Ziegler to argue that the primes too much contain arbitrarily long polynomial progressions.

(The above account is not completely accurate, but it is not lying too much.)

As announced last October here, and here, Omer Reingold, Madhur Tulsiani, Salil Vadhan and I found a new proof of this “dense model” theorem, which uses the min-max theorem of game theory (or, depending on the language that you prefer to use, the duality of linear programming or the Hahn Banach theorem) and was inspired by Nisan’s proof of the Impagliazzo hard-core set theorem. In complexity-theoretic applications of the theorem, our reduction has polynomial complexity, while the previous work incurred an exponential loss.

As discussed here and here, we also show how to use the Green-Tao-Ziegler techniques of “iterative partitioning” to give a different proof of Impagliazzo’s theorem.

After long procrastination, we recently wrote up a paper about these results.

In the Fall, we received some feedback from additive combinatorialists that while our proof of the Green-Tao-Ziegler result was technically simpler than the original one, the language we used was hard to follow. (That’s easy to believe, because it took us a while to understand the language in which the original proof was written.) We then wrote an expository note of the proof in the analyst’s language. When we were about to release the paper and the note, we were contacted by Tim Gowers, who, last Summer, had independently discovered a proof of the Green-Tao-Ziegler results via the Hahn-Banach theorem, essentially with the same argument. (He also found other applications of the technique in additive combinatorics. The issue of polynomial complexity, which does not arise in his applications, is not considered.)

Gowers announced his results in April at a talk at the Fields institute in Toronto. (Audio and slides are available online.)

Gowers’ paper already contains the proof presented in the “analyst language,” making our expository note not so useful any more; we have still posted it anyways because, by explaining how one translates from one notation to the other, it can be a short starting point for the computer scientist who is interested in trying to read Gowers’ paper, or for the combinatorialist who is interested in trying to read our paper.

Bourgain’s Embedding of Any Metric into L1

June 5, 2008 in math, theory | Tags: Expanders, Metric embeddings | | 1 comment

In which we spend more than sixteen hundred words to explain a three-line proof.

In the last post, we left off with the following problem. We have a set $V$ of $n$ “vertices,” a semi-metric $d: V\times V \rightarrow {\mathbb R}$ , and we want to find a distribution over sets $A\subseteq V$ such that for every two vertices $i,j$

$(1) \ \ \ \displaystyle {\mathbb E} | d(A,i) - d(A,j) | \geq \frac 1 {O(\log n)} \cdot d(i,j)$

where

$\displaystyle d(A,i) := \min_{a\in A} d(a,i)$

This will give us a way to round a solution of the Leighton-Rao relaxation to an actual cut with only an $O(\log n)$ loss in the approximation.

Before getting to the distribution which will do the trick, it is helpful to consider a few examples.

Example 1: all points are at distance 1 from each other.
Then $|d(A,i)-d(A,j)|$ is equal to either 0 or 1, and it is 1 if and only if $A$ contains exactly one of $i$ or $j$ . If $A$ is a uniformly chosen random set, then the above condition is satisfied with probability $1/2$ , so we have the stronger bound

$\displaystyle {\mathbb E} | d(A,i) - d(A,j) | \geq \frac 12 \cdot d(i,j)$

[Indeed, even better, we have ${\mathbb E} | d(A,i) - d(A,j) | = \frac 12 \cdot d(i,j)$ , which is an isometric embedding.]
Example 2: all points are at distance either 1 or 2 from each other.
If $A$ contains exactly one of the vertices $i,j$ , then $|d(A,i)-d(A,j)| \geq 1$ , and so, if we choose $A$ uniformly at random we have

$\displaystyle {\mathbb E} | d(A,i) - d(A,j) | \geq \frac 14 \cdot d(i,j)$

These examples may trick us into thinking that a uniformly chosen random set always work, but this unfortunately is not the case.

Example 3: Within a set $S$ of size $n/2$ , all distances are 1, and the same is true within $V-S$ ; the distance between elements of $S$ and elements of $V-S$ is $n$ .
If we consider $i \in S$ and $j \in V-S$ , then we are in trouble whenever $A$ contains elements from both sets, because then $|d(A,i)-d(A,j)| \leq 1$ , while $d(i,j)=n$ . If we pick $A$ uniformly at random, then $A$ will essentially always, except with exponentially small probability, contain elements from both $S$ and $V-S$ . If, however, we pick $A$ to be a random set of size 1, then we are going to get $|d(A,i)-d(A,j)| \geq n-2$ with probability at last $1/2$ , which is great.

Choosing a set of size 1, however, is a disaster inside $S$ and inside $V-S$ , where almost all distances $|d(A,i)-d(A,j)|$ collapse to zero. For those pairs, however, we know that choosing $A$ uniformly at random works well.

The solution is thus: with probability 1/2, pick $A$ uniformly at random; with probability 1/2, pick $A$ of size 1.

So far we are actually getting away with ${\mathbb E} |X_i - X_j|$ being a constant fraction of $d(i,j)$ . Here is a slightly trickier case.

Example 4: The shortest path metric in a $\sqrt n \times \sqrt n$ grid.
Take two vertices $i,j$ at distance $d$ . We can get, say, $|d(A,i) - d(A,j)| \geq d/3$ , provided that $A$ avoids all vertices at distance $\leq 2d/3$ from $i$ , and it includes some vertex at distance $\leq d/3$ from $j$ . In a grid, the number of vertices at distance $\leq d$ from a given vertex is $\Theta(d^2)$ , so our goal is to pick $A$ so that it avoids a certain set of size $\Theta((2d/3)^2) = \Theta(d^2)$ and it hits another set of size $\Theta( (d/3)^2) = \Theta(d^2)$ . If we pick $A$ to be a random set of size about $n/d^2$ , both events hold with constant probability.

Now, what works for a certain distance won’t work for a different distance, so it seems we have to do something like picking $t$ from $1$ to $n$ , and then pick a random set $A$ of size $n/t$ . This is however too bad, because our chance of getting a set of the right size would only be $1/n$ , while we can only lose a factor $1/\log n$ . The solution is to pick $t$ at random from $\{ 1,2,4,\cdots,n\}$ , and then pick $A$ of size $A/t$ . With probability $1/\log n$ we get the right size of $A$ , up to a factor of two.

It turns out that the last example gives a distribution that works in all cases:

Pick $t$ at random in $1 \cdots \log n$
Pick a random set $A$ so that each $i\in V$ is selected to be in $A$ independently and with probability $1/2^t$

Now, it would be nice to show that (as in the examples we have seen so far) for every semi-metric $d$ and two vertices $i,j$ , there is a size parameter $t$ such that when $A$ is chosen to be a random set of size $n/2^t$ we have ${\mathbb E} |d(A,i)-d(A,j)| \geq \Omega(d(i,j))$ .

This would mean that, after we lose a factor of $\log n$ to “guess” the right density, we have the desired bound (3). Unfortunately this is too much to ask for; we shall instead work out an argument that uses contributions from all densities.

It is good to see one more example.

Example 5: A 3-regular graph of logarithmic girth.
Let $i,j$ be two vertices whose distance $d$ is less than the girth. In the example of the grid, we considered all vertices at distance $\leq 2d/3$ from $i$ and all vertices at distance $\leq d/3$ from $j$ ; in this case, there are $\approx 2^{2d/3}$ of the former, and $\approx 2^{d/3}$ of the latter, and it is hopeless to expect that $A$ , no matter its density, can avoid all of the former, but hit some of the latter.

If, however, I consider the points at distance $\leq \ell+1$ from $i$ and the points at distance $\leq \ell$ from $j$ , they are off only by a constant factor, and there is a constant probability of avoiding the former and hitting the latter when $t = \ell$ . So conditioned on each $t$ between $1$ and $d/2$ , the expectation of $|d(A,i) - d(A,j)|$ is at least $\Omega(1)$ , and, overall, the expectation is at least $\Omega(d/\log n)$ .

We are now more or less ready to tackle the general case.

We look at two vertices $i,j$ at distance $d(i,j)$ and we want to estimate ${\mathbb E} |d(A,i)-d(A,j)|$ .

Let us estimate the contribution to the expectation coming from the case in which we choose a particular value of $t$ . In such a case, there is a constant probability that

$A$ contains none of the $2^{t+1}-1$ vertices closest to $i$ , but at least one of the of $2^t$ vertices closest to $j$ . [Assuming the two sets are disjoint]
$A$ contains none of the $2^{t+1}-1$ vertices closest to $j$ , but at least one of the of $2^t$ vertices closest to $i$ . [Assuming the two sets are disjoint]

Notice that events (1) and (2) are disjoint, so we are allowed to sum their contributions to the expectation, without doing any double-counting.

Call $ri_k$ the distance of the the $k$ -th closest vertex from $i$ , and similarly $rj_k$ . Then, if event (1) happens, $d(A,i) \geq ri_{2^{t+1}}$ and $d(A,j) \leq rj_{2^t}$ , in which case

$|d(A,i) - d(A,j)| \geq d(A,i) - d(A,j) \geq ri_{2^{t+1}} - rj_{2^t}$

we can similarly argue that if (2) happens, then

$|d(A,i) - d(A,j)| \geq d(A,j) - d(A,i) \geq rj_{2^{t+1}} - ri_{2^t}$

Call $r'i_{k} := \min \{ d(i,j)/3, ri_k\}$ and $r'j_{k} := \min \{ d(i,j)/3, rj_k\}$ .

Let $t^*$ be the smallest $t$ such that either $ri_{2^{t+1}}\geq d(i,j)/3$
or $rj_{2^{t+1}}\geq d(i,j)/3$ . Then for $t\leq t^*-1$ the events described in (1) and (2) are well-defined, and the contributions to the expectation is at least

$\frac 1 {\log n} \cdot ( ri_{2^{t+1}} - rj_{2^t} + rj_{2^{t+1}} - ri_{2^t} ) =$
$= \frac 1 {\log n} \cdot ( r'i_{2^{t+1}} - r'j_{2^t} + r'j_{2^{t+1}} - r'i_{2^t} )$

When $t=t^*$ , we can verify that the contribution to the expectation is at least

$= \frac 1 {\log n} \cdot ( r'i_{2^{t^*+1}} - r'j_{2^{t^*}} + r'j_{2^{t^*+1}} - r'i_{2^{t^*}} )$
$\geq \frac 1 {\log n} \cdot ( d(i,j)/3 - r'j_{2^{t^*}} - r'i_{2^{t^*}}$

And if we sum the contributions for $t=0,\cdots t^*$ , the sum telescopes and we are left with

$\displaystyle {\mathbb E} |d(A,i) - d(A,j) | \geq \Omega\left( \frac 1 {\log n} \right) \cdot \left( \frac{d(i,j)}{3} - r'i_{1} - r'j_1 \right)$

where $r'i_1 = r'j_1 = 0$ .

At long last, we have completed the proof.

Notice that the $O(\log n)$ factor we have lost is best possible in light of the expander example we saw in the previous post. In many examples, however, we lost only a constant factor. It is a great open question whether it is possible to lose only a constant factor whenever the metric is a shortest-path metric on a planar graph.

How Good is the Leighton-Rao Relaxation?

June 4, 2008 in math, theory | Tags: Expanders, Integrality gap, Metric embeddings | | No comments

Continuing our seemingly never-ending series on approximations to edge expansion, we come to the question of how good is the Leighton-Rao relaxation.

Given a graph $G=(V,E)$ , we are trying to approximate the sparsest cut problem (which, in turn, approximates the edge expansion problem within a factor of two) defined as

$\displaystyle \phi(G) := \min_{S\subseteq V} \frac {edges(S,V-S)} {\frac 1n |S| \cdot |V-S|}$

And we have seen that an equivalent characterization of sparsest cut is:

$(1) \ \ \ \displaystyle \phi(G) = \min_{x\in {\mathbb R}^V} \frac{ \sum_{i,j} A(i,j) | x_i - x_j | } {\frac 1n \sum_{i,j} |x_i - x_j | }$

In the Leighton-Rao relaxation, we have

$(2) \ \ \ \displaystyle LR(G) := \min_{d\in {\cal M}(V)} \frac{ \sum_{i,j} A(i,j) d(i,j) }{\frac 1n \sum_{i,j} d(i,j)}$

where ${\cal M}(V)$ is the set of all semi-metrics $d: V \times V \rightarrow {\mathbb R}$ , that is, of all functions such that $d(i,j) = d(j,i) \geq 0$ , $d(i,i)=0$ , and that obey the “triangle inequality”

$d(i,j) \leq d(i,k) + d(k,j)$

for all $i,j,k \in V$ .

This is clearly a relaxation of the sparsest cut problem, because for every $x \in {\mathbb R}^V$ the “distances” $|x_i - x_j|$ define indeed a semi-metric over $V$ .

How bad can the relaxation be? Take a family of regular constant-degree expander graphs, that is graphs where $\phi$ is a constant and the degree $k$ is a constant independent of $n=|V|$ . Define $d(i,j)$ to be the shortest path distance between $i$ and $j$ in the graph. This is clearly a semi-metric, and we have

$\displaystyle \sum_{i,j} A(i,j) d(i,j) = kn = O(n)$

because along every edge the distance is precisely one, and

$\displaystyle \sum_{i,j} d(i,j) = \Omega (n^2 \log_k n)$

because, for every vertex $i$ , at most $k^t$ other vertices can be within
distance $t$ , and so at least half the vertices are at distance $\Omega( \log_k n)$ .

So we have

$\displaystyle LR(G) = O \left( \frac 1 {\log n} \right)$

even though $\phi(G) = \Omega(1)$ , so the error in the approximation can be as bad as a factor of $\log n$ . We shall prove that this is as bad as it gets.

Our task is then: given a feasible solution to (2), that is, a distance function defined on the vertices of $G$ , find a feasible solution to sparsest cut whose cost is as small as possible, and no more than a $O(\log n)$ factor off from the cost of the solution to (2) that we started from. Instead of directly looking for a cut, we shall look for a solution to (1), which is a more flexible formulation.

The general strategy will be, given $d$ , to find a distribution $X$ over vectors $x \in {\mathbb R}^V$ such that for every two vertices $i,j$ we have

$(3) \ \ \ \displaystyle \Omega\left( \frac {1}{\log n} \right) \cdot d(i,j) \leq {\mathbb E} | X_i - X_j | \leq d(i,j)$

Suppose that we start from a solution to (2) of cost $c$ , that is, we have a semimetric $d(\cdot,\cdot)$ such that

$\displaystyle \sum_{i,j} A_{i,j} d(i,j) = c \cdot \frac 1n \sum_{i,j} d(i,j)$

Then our distribution $X$ is such that

$\displaystyle {\mathbb E} \sum_{i,j} A_{i,j} |X_i - X_j| \leq \sum_{i,j} A_{i,j} d(i,j)$

and

$\displaystyle \sum_{i,j} d(i,j) \leq O \left( {\log n} \right) {\mathbb E} \sum_{i,j}|X_i - X_j|$

$\displaystyle {\mathbb E} \left[ \sum_{i,j} A_{i,j} |X_i - X_j| - O(c\log n) \cdot \frac 1n \sum_{i,j} |X_i - X_j| \right] \leq 0$

and there must exist a vector $x$ in the support of the distribution $X$ such that

$\displaystyle \sum_{i,j} A_{i,j} |x_i - x_j| - O(c\log n) \cdot \frac 1n \sum_{i,j} |x_i - x_j| \leq 0$

and now we are done because the cost of $x$ in (1) is at most $O(\log n)$ times the optimum of the Leighton-Rao relaxation.

Though this works, it seems an overkill to require condition (3) for all pairs of vertices. It seems sufficient to require

${\mathbb E} |X_i-X_j| \leq d(i,j)$

only for edges, and to require

$d(i,j) \leq O(\log n) \cdot {\mathbb E} |X_i-X_j|$

only “on average” over all pairs of vertices. It can be shown, however, that if one wants to round a generalization of the Leighton-Rao relaxation to sparsest cut with “non-uniform demands,” then any rounding algorithm can be turned into a rounding algorithm that satisfies (3).

So how do we start from an arbitrary metric and come up with a probabilistic linear order of the vertices so that their distances in the original metric are well approximated in the linear ordering? We will describe a method of Bourgain, whose applicability in this setting is due Linial, London and Rabinovich.

The starting point is to see that if $A$ is a set of vertices, and we define

$\displaystyle X_i := d(A,i) := \min_{a\in A} d(a,i)$

then, no matter how we choose $A$ , we always have

$|X_i - X_j | = |d(A,i) - d(A,j) | \leq d(i,j)$

[Hint: use the triangle inequality to see that $d(A,j) \leq d(A,i) + d(i,j)$ .]

This means that “all we have to do” is to find a distribution over sets $A$ such that for every $i,j$ we have

$(3) \ \ \ \displaystyle {\mathbb E} | d(A,i) - d(A,j) | \geq \Omega \left( \frac 1 {\log n} \right) \cdot d(i,j)$

If you want to make it sound more high-brow, you may call such a distribution a Frechet embedding of $d$ into $\ell_1$ . Constructing such a distribution will be the subject of the next post.

The Limitations of the Spectral Partitioning Algorithm

May 23, 2008 in math, theory | Tags: Expanders, Integrality gap, Spectral partitioning | | No comments

We continue to talk about the problem of estimating the expansion of a graph $G=(V,E)$ , focusing on the closely related sparsest cut, defined as

$\displaystyle \phi(G):= \min_{S} \frac {edges(S,V-S) } { \frac 1n \cdot |S| \cdot |V-S| }$

The spectral paritioning algorithm first finds a vector $x \in {\mathbb R}^n$ minimizing

$(1) \ \ \ \displaystyle \frac{ \sum_{i,j} A(i,j) |x(i)-x(j)|^2 } { \frac 1n \cdot \sum_{i,j} |x(i)-x(j)|^2}$

(where $A$ is the adjacency matrix of $G$ ) and then finds the best cut $(S,V-S)$ where $S$ is of the form $\{ i \in V: \ x(i) \geq t \}$ for a threshold $t$ .

We proved that if the quantity in (1) is $\epsilon$ and $G$ is $d$ -regular, then the algorithm will find a cut of sparsity at most $\sqrt{8 \epsilon d}$ , and that if $x$ is the eigenvector of the second eigenvalue, then it is an optimal solution to (1), and the cost of an optimal solution to (1) is a lower bound to $\phi(G)$ . This means that the algorithm finds a cut of sparsity at most $\sqrt{8 \phi(G) d}$ .

Can the analysis be improved? Read the rest of this entry »

Giving American Flags and Conceptual Contributions to Orphans

May 22, 2008 in philosophy, theory | Tags: Conceptual contributions | | 17 comments

I am worried that, lately, I am agreeing too much with James Lee. I hope my friends will stage an intervention if I start saying things like “…this group acts by isometries on Hilbert space…”

I agree with his choice of favorite STOC’08 talks, but also with his comments on the statement on the importance of conceptual contributions in theoretical computer science, which was recently written by a group of distinguished theoreticians, was briefly discussed at the STOC’08 business meeting, and was commented upon here, here and here.

In Mr. Spritz Goes to Washington, Lisa and Homer help pass a bill to divert flights from over their house by tacking their air traffic bill on a “giving orphans American flags” bill.

I felt that the statement (and the subsequent public and private discussions) similarly mixes the unobjectionable with the potentially controversial. There have been two themes that are entirely agreeable.

One is the importance, and value, of simplicity in proofs. This goes without saying, and I think that the community does a good job in recognizing simple resolutions of long-standing open questions, as well as new simplified proofs of known results. In fact, I think it would be hard to find someone who suggests that, all other things being equal, a harder proof is preferable to a simple proof.

Another is the importance of introducing new concepts and models. Indeed, there would be no theoretical computer science without its “concepts,” and the field would die if it would stop innovating. Again, nobody could possibly argue against the importance of new definitions and models.

Then, there is something which is said in the statement in a way that is not self-evident:

Once understood, conceptual aspects tend to be viewed as obvious, which actually means that they have become fully incorporated in the worldview of the expert. This positive effect is actually a source of trouble in the evaluation process, because the evaluators forget that these contributions were not obvious at all before being made.

Indeed, our community should be warned of dismissing such contributions by saying “yes, but that’s obvious”; when somebody says such a thing, one should ask “was it obvious to you before reading this article?”

I think (and I echo things that were better said in comments that I linked to above) that if something looks obvious after reading a new paper for the first time (as opposed to looking back at a classical paper from twenty or more years ago), then the paper is not making a valuable conceptual contribution. The time that it takes to read a paper cannot be sufficient to “alter the worldview” of the reader: the concept must have been “in the air.”

To be sure, usually even the greatest discoveries are in the air when they are made, a point which I want to write about in a different post. The fundamental difference, however, is that reading about a good conceptual discovery for the first time is startling and exciting, like hearing a good joke for the first time. If an expert feels nothing when reading about a conceptual discovery, it is usually a very bad sign, although one can come up with various exceptions.

Often quoted exceptions are some early papers in the great foundations-of-cryptography revolution of 1982, especially the Goldwasser-Micali-Rackoff paper on Zero Knowledge, which ended up appearing in 1985. What was special about that line of work is that it was not in the air, but, rather, ahead of its time. Obviously, I wasn’t there, so I don’t know, but I guess that people at the time were not saying “this work is obvious,” but rather “what the hell is this?,” which is quite different.

A final remark is that when a paper presents a new model or problem (and I understand that these are the kind of papers that the statement refers to), there can be no “intrinsic” value attributed the model or the problem; the value will be in the understanding and general progress that will come by studying the model and finding solutions to the problem, and how this will connect with different problems, different models, and applications.

There seem to be only two ways to validate a new conceptual proposal: one is to present preliminary evidence that one can do interesting things with it. (I think, in this case, too much emphasis is put on achieving quantitative improvement; I am impressed enough to see known, hard, results, recovered in a completely different way.) The other is the “gut feeling” of the expert, which feels that the new ideas are the “right” way to think about the issue at hand.

It seems right to reject a paper that fails both tests.

The Spectral Partitioning Algorithm

May 11, 2008 in math, theory | Tags: Expanders, Spectral partitioning | | 4 comments

[In which we prove the "difficult part" of Cheeger's inequality by analyzing a randomized rounding algorithm for a continuous relaxation of sparsest cut.]

We return to this month’s question: if $G=(V,E)$ is a $d$ -regular graph, how well can we approximate its edge expansion $h(G)$ defined as

$h(G) := \min_{S\subseteq V} \frac{edges(S,V-S)} {\min \{ |S|, \ |V-S| \} }$

and its sparsest cut $\phi(G)$ defined as

$\phi(G) := \min_{S\subseteq V} \frac{edges(S,V-S)} { \frac 1n \cdot |S| \cdot |V-S|} = \min_{x\in \{0,1\}^n } \frac{ \sum_{i,j} A(i,j) \cdot |x(i)-x(j)|}{\frac 1n \sum_{i,j} |x(i)-x(j)|}$ ,

where $A$ is the adjacency matrix of $G$ .

We have looked at three continuous relaxations of $\phi(G)$ , the spectral gap, the Leighton-Rao linear program, and the Arora-Rao-Vazirani semidefinite program.

As we saw, the spectral gap of $G$ , defined as the difference between largest and second largest eigenvalue of $A$ , can be seen as the solution to a continuous optimization problem:

$d-\lambda_2 = \min_{x\in {\mathbb R}^n} \frac {\sum_{i,j} A(i,j) \cdot |x(i)-x(j)|^2}{\frac 1n \sum_{i,j} |x(i)-x(j)|^2}$ .

It follows from the definitions that

$d-\lambda_2 \leq \phi \leq 2h$

which is the “easy direction” of Cheeger’s inequality, and the interesting thing is that $d-\lambda_2$ is never much smaller, and it obeys

$(1) \ \ \ d-\lambda_2 \geq \frac {h^2}{2d} \geq \frac {\phi^2}{8d}$ ,

which is the difficult part of Cheeger’s inequality. When we normalize all quantities by the degree, the inequality reads as

$\frac 1 8 \cdot \left( \frac \phi d \right)^2 \leq \frac 1 2 \cdot \left( \frac h d \right)^2 \leq \frac{d-\lambda_2}{d} \leq \frac \phi d \leq 2 \frac h d$ .

I have taught (1) in three courses and used it in two papers, but I had never really understood it, where I consider a mathematical proof to be understood if one can see it as a series of inevitable steps. Many steps in the proofs of (1) I had read, however, looked like magic tricks with no explanation. Finally, however, I have found a way to describe the proof that makes sense to me. (I note that everything I will say in this post will be completely obvious to the experts, but I hope some non-expert will read it and find it helpful.)

We prove (1), as usual, by showing that given any $x\in {\mathbb R}^n$ such that

$(2) \ \ \ \frac {\sum_{i,j} A(i,j) \cdot |x(i)-x(j)|^2}{\frac 1n \sum_{i,j} |x(i)-x(j)|^2} = \epsilon$

we can find a threshold $t$ such that the cut $(S,V-S)$ defined by $S:= \{ i: x_i \geq t \}$ satisfies

$(3) \ \ \ \frac{edges(S,V-S)} { \min \{ |S|,\ |V-S| \} } \leq \sqrt{2 d \epsilon}$

and

$\frac{edges(S,V-S)} { \frac 1n \cdot |S| \cdot |V-S| } \leq \sqrt{8 d \epsilon}$ .

This not only gives us a proof of (1), but also an algorithm for finding sparse cuts when they exist: take a vector $x$ which is an eigenvector of the second eigenvalue (or simply a vector for which the Rayleigh quotient in (2) is small), sort the vertices $i$ according to the value of $x(i)$ , and find the best cut among the “threshold cuts.” This is the “spectral partitioning” algorithm.

This means that proving (1) amounts to studying an algorithm that “rounds” the solution of a continuous relaxation to a combinatorial solution, and there is a standard pattern to such arguments in computer science: we describe a randomized rounding algorithm, study its average performance, and then argue that there is a fixed choice that is at least as good as an average one. Here, in particular, we would like to find a distribution $T$ over threshold, such that if we define $S:= \{ i: x(i) \geq T\}$ as a random variable in terms of $T$ we have

${\mathbb E} [ edges(S,V-S) ] \leq {\mathbb E} [\min \{ |S|,\ |V-S| \} ] \cdot \sqrt{2 d \epsilon}$

and so, using linearity of expectation,

${\mathbb E} [ edges(S,V-S) - \min \{ |S|,\ |V-S| \} \cdot \sqrt{2 d \epsilon}] \leq 0$

from which we see that there must be a threshold in our sample space such that (3) holds. I shall present a proof that explicitly follows this pattern. It would be nice if we could choose $T$ uniformly at random in the interval $\left[\min_i x(i), \max_i x(i)\right]$ , but I don’t think it would work. (Any reader can see a counterexample? I couldn’t, but we’ll come back to it.) Instead, the following works: assuming with no loss of generality that the median of $\{ x(1),\ldots,x(n) \}$ is zero, $T$ can be chosen so that $|T|\cdot T$ is distributed uniformly in the interval $\left[\min_i x(i), \max_i x(i)\right]$ . (This means that thresholds near the median are less likely to be picked than thresholds far from the median.) This choice seems to be a magic trick in itself, voiding the point I made above, but I hope it will become natural as we unfold our argument. Read the rest of this entry »

Relaxations of Edge Expansion and Their Duals (Part 2)

May 5, 2008 in math, theory | Tags: Expanders | | 1 comment

[In which we finally get to Leighton-Rao and ARV.]

Continuing our edge expansion marathon, we are going to see more ways in which the sparsest cut problem can be relaxed to polynomial-time solvable continuous problems. As before, our goal is to understand how this implies certificates of expansion.

We are talking about the sparsest cut $\phi(G)$ of a graph $G(V,E)$ , which is

$\phi(G) := \min_{S\subseteq V} \frac{edges(S,V-S)}{\frac 1n |S|\cdot V-S|} = \min_{x\in \{0,1\}^n} \frac { \sum_{i,j} A(i,j) |x(i)-x(j)|} { \frac 1n \sum_{i,j} |x(i)-x(j)|}$

We have looked for a while at the eigenvalues gap of $G$ . If $G$ is $d$ -regular and the second largest eigenvalue of $G$ is $\lambda_2$ , then the difference between largest and second largest eigenvalue obeys

$d-\lambda_2 = \min_{x\in {\mathbb R}^n} \frac { \sum_{i,j} A(i,j) |x(i)-x(j)|^2} {\frac 1n \sum_{i,j} |x(i)-x(j)|^2} = \min_{m,x_1,\ldots,x_n\in {\mathbb R}^m} \frac { \sum_{i,j} A(i,j) ||x_-x_j||^2} {\frac 1n \sum_{i,j} ||x_i-x_j||^2}$

and, noting that $|x(i)-x(j)|^2 = |x(i)-x(j)|$ when $x\in \{0,1\}^n$ , we saw that $d-\lambda_2$ is a relaxation of $\phi$ , and $\phi \geq d-\lambda_2$ .

Leighton and Rao derive a continuous relaxation of $\phi$ in quite a different way. They note that when $x\in \{ 0,1\}^n$ , then the “distance”

$d(i,j) := |x(i)-x(j)|$

induces a semi-metric on $V$ , that is, $d(i,i)=0$ , $d(i,j) = d(j,i)$ , and the triangle inequality holds

$d(i,j) \leq d(i,k) + d(k,j) \ \ \ \forall i,j,k$

One can then consider the relaxation

$(1) \ \ \ min_d \frac { \sum_{i,j} A(i,j) d(i,j)} {\frac 1n \sum_{i,j} d(i,j)}$

where the minimum is taken over all distance functions $d: V\times V \rightarrow {\mathbb R}$ that define a semi-metric on $V$ .

We can rewrite (1) as the linear program

$(2) \ \ \ \begin{array}{l} \min \sum_{i,j} A(i,j) d(i,j)\\ subject\ to \\ \sum_{i,j} d(i,j)=n\\ d(i,j) \leq d(i,k) + d(k,j) \ \ \ \forall i,j,k\\ d(i,j) \geq 0 \ \ \ \forall i,j \end{array}$

where we only write the triangle inequality constraints by having one variable $d(i,j)$ for every unordered pair $i,j$ .

The minimum of (2) gives a lower bound to the sparsest cut of $G$ , and so every feasible solution for the dual of (2) gives a lower bound to $\phi(G)$ .

Before trying to write down the dual of (2), it is convenient to write (2) in a less efficient way. (This is akin to the fact that sometimes, when using induction, it is easier to prove a more general statement.) Instead of writing the triangle inequality for every three vertices, we write it for every arbitrary sequence of vertices.

$(3) \ \ \ \begin{array}{l} \min \sum_{i,j} A(i,j) d(i,j)\\ subject\ to \\ \sum_{i,j} d(i,j)=n\\ d(i,j) \leq \sum_{(u,v) \in p} d(u,v) \ \ \ \forall i,j, \forall p \in P_{i,j} \\ d(i,j) \geq 0 \ \ \ \forall i,j\end{array}$

where we denote by $P_{i,j}$ the set of “paths” in the complete graph between $i$ and $j$ . (That is, $P_{i,j}$ is the set of all sequences of vertices that start at $i$ and end at $j$ .)

Clearly (3) is the same as (2), in the sense that the extra inequalities present in (3) are implied by the triangle inequalities in (2). The dual of (3), however, is cleaner, and is given below

$(4) \ \ \ \begin{array}{l} \max n\cdot y\\ \ y - \sum_{p\in P_{i,j}} y_p + \sum_{p: (i,j) \in p} y_p \leq A(i,j)\ \ \ \forall i,j\\ y,y_p \geq 0\end{array}$

A feasible solution to (4) is thus a weighted set of paths and a parameter $y$ . Before reasoning about the meaning of a solution, suppose that $y,y_p$ satisfy the stronger constraints

$(5) \ \ \ \begin{array}{l} \sum_{p\in P_{i,j}} y_p \geq y\\\sum_{p: (i,j) \in p} y_p \leq A(i,j) \end{array}$

Then we can view the $y_p : p \in P_{i,j}$ as defining a flow between $i,j$ that carries at least $y$ units of flow; furthermore, the union of all such flows uses at most a total capacity $A(i,j)$ on each edge $(i,j)$ . We should think of it as an embedding of the complete graph scaled by $y$ into $G$ . I want to claim that this means that the sparsest cut of $G$ is at least $y$ times the sparsest cut of the complete graph, that is, at least $y\cdot n$ .

To verify the claim, take any cut $S,V-S$ . We know that such a cut separates $|S| \cdot |V-S|$ pairs of vertices $i,j$ , and that at least $y$ units of flow are routed between each such pair, so that at least $y\cdot |S| \cdot |V-S|$ units of flow cross the cut. Since every edge has capacity 1, we see that at least $y\cdot |S|\cdot |V-S|$ edges of $G$ must cross the cut.

It remains to bridge the difference between (4) or (5). One possibility is to show that a solution to (4) can be modified to a solution satisfying (5) with no loss in $y$ . Another approach is to directly use (4) and a general weak duality argument to show directly that a solution satisfying (4) implies that $G$ has at least the edge expansion of a complete graph with edge weights $y$ .

There is, in fact, a broader principle at work, which we can verify by a similar argument: if $\{ y_p\}$ is an assignment of weights to all possible paths (which we think of as defining a multicommodity flow), $G$ is a graph with adjacency matrix $A$ , $H$ is a graph with adjacency matrix $B$ , and we have

$B(i,j) - \sum_{p\in P_{i,j}} y_p + \sum_{p: (i,j) \in p} y_p \leq A(i,j)\ \ \ \forall i,j$

then we must have $\phi(G) \geq \phi(H)$ .

The fact that a feasible solution to (4) gives a lower bound to $\phi(G)$ is a special case that applies to the case in which $H$ is a scaled version of the complete graph.

This is a good point to pause, look back again at the bound coming from spectral considerations, and compare it with what we have here. Read the rest of this entry »

Relaxations of Edge Expansion and Their Duals

May 3, 2008 in math, theory | Tags: Expanders | | 5 comments

In the previous post, we saw that the sparsest cut of a graph $G=(V,E)$ (which is within a factor of 2 of the edge expansion of the graph), is defined as

$\phi := \min_{S\subseteq V} \frac{edges(S,V-S)}{\frac 1n |S|\cdot |V-S|}$

and can also be written as

$(1) \ \ \ \phi = \min_{x\in \{0,1\}^V} \frac{\sum_{i,j} A(i,j) | x(i)-x(j)|}{\frac 1n \sum_{i,j} |x(i)-x(j)|}$

while the eigenvalue gap of $G$ , can be written as