in theory http://lucatrevisan.wordpress.com "Marge, I agree with you - in theory. In theory, communism works. In theory." -- Homer Simpson Mon, 12 May 2008 04:25:10 +0000 http://wordpress.org/?v=MU en The Spectral Partitioning Algorithm http://lucatrevisan.wordpress.com/2008/05/11/the-spectral-partitioning-algorithm/ http://lucatrevisan.wordpress.com/2008/05/11/the-spectral-partitioning-algorithm/#comments Sun, 11 May 2008 20:00:15 +0000 luca http://lucatrevisan.wordpress.com/?p=400

[In which we prove the "difficult part" of Cheeger's inequality by analyzing a randomized rounding algorithm for a continuous relaxation of sparsest cut.]

We return to this month’s question: if G=(V,E) is a d-regular graph, how well can we approximate its edge expansion h(G) defined as

h(G) := \min_{S\subseteq V} \frac{edges(S,V-S)} {\min \{ |S|, \ |V-S| \} }

and its sparsest cut \phi(G) defined as

\phi(G) := \min_{S\subseteq V} \frac{edges(S,V-S)} { \frac 1n \cdot |S| \cdot |V-S|} = \min_{x\in \{0,1\}^n } \frac{ \sum_{i,j} A(i,j) \cdot |x(i)-x(j)|}{\frac 1n \sum_{i,j} |x(i)-x(j)|},

where A is the adjacency matrix of G.

We have looked at three continuous relaxations of \phi(G), the spectral gap, the Leighton-Rao linear program, and the Arora-Rao-Vazirani semidefinite program.

As we saw, the spectral gap of G, defined as the difference between largest and second largest eigenvalue of A, can be seen as the solution to a continuous optimization problem:

d-\lambda_2 = \min_{x\in {\mathbb R}^n} \frac {\sum_{i,j} A(i,j) \cdot |x(i)-x(j)|^2}{\frac 1n \sum_{i,j} |x(i)-x(j)|^2}.

It follows from the definitions that

d-\lambda_2 \leq \phi \leq 2h

which is the “easy direction” of Cheeger’s inequality, and the interesting thing is that d-\lambda_2 is never much smaller, and it obeys

(1) \ \ \ d-\lambda_2 \geq \frac {h^2}{2d} \geq \frac {\phi^2}{8d},

which is the difficult part of Cheeger’s inequality. When we normalize all quantities by the degree, the inequality reads as

\frac 1 8 \cdot \left( \frac \phi d \right)^2 \leq \frac 1 2 \cdot \left( \frac h d \right)^2 \leq \frac{d-\lambda_2}{d} \leq \frac \phi d \leq 2 \frac h d .

I have taught (1) in three courses and used it in two papers, but I had never really understood it, where I consider a mathematical proof to be understood if one can see it as a series of inevitable steps. Many steps in the proofs of (1) I had read, however, looked like magic tricks with no explanation. Finally, however, I have found a way to describe the proof that makes sense to me. (I note that everything I will say in this post will be completely obvious to the experts, but I hope some non-expert will read it and find it helpful.)

We prove (1), as usual, by showing that given any x\in {\mathbb R}^n such that

(2) \ \ \ \frac {\sum_{i,j} A(i,j) \cdot |x(i)-x(j)|^2}{\frac 1n \sum_{i,j} |x(i)-x(j)|^2} = \epsilon

we can find a threshold t such that the cut (S,V-S) defined by S:= \{ i: x_i \geq t \} satisfies

(3) \ \ \ \frac{edges(S,V-S)} { \min \{ |S|,\ |V-S| \} } \leq \sqrt{2 d \epsilon}

and

\frac{edges(S,V-S)} { \frac 1n \cdot |S| \cdot |V-S| } \leq \sqrt{8 d \epsilon}.

This not only gives us a proof of (1), but also an algorithm for finding sparse cuts when they exist: take a vector x which is an eigenvector of the second eigenvalue (or simply a vector for which the Rayleigh quotient in (2) is small), sort the vertices i according to the value of x(i), and find the best cut among the “threshold cuts.” This is the “spectral partitioning” algorithm.

This means that proving (1) amounts to studying an algorithm that “rounds” the solution of a continuous relaxation to a combinatorial solution, and there is a standard pattern to such arguments in computer science: we describe a randomized rounding algorithm, study its average performance, and then argue that there is a fixed choice that is at least as good as an average one. Here, in particular, we would like to find a distribution T over threshold, such that if we define S:= \{ i: x(i) \geq T\} as a random variable in terms of T we have

{\mathbb E} [ edges(S,V-S) ] \leq {\mathbb E} [\min \{ |S|,\ |V-S| \} ] \cdot \sqrt{2 d \epsilon}

and so, using linearity of expectation,

{\mathbb E} [ edges(S,V-S) - \min \{ |S|,\ |V-S| \} \cdot \sqrt{2 d \epsilon}] \leq 0

from which we see that there must be a threshold in our sample space such that (3) holds. I shall present a proof that explicitly follows this pattern. It would be nice if we could choose T uniformly at random in the interval \left[\min_i x(i), \max_i x(i)\right], but I don’t think it would work. (Any reader can see a counterexample? I couldn’t, but we’ll come back to it.) Instead, the following works: assuming with no loss of generality that the median of \{ x(1),\ldots,x(n) \} is zero, T can be chosen so that |T|\cdot T is distributed uniformly in the interval \left[\min_i x(i), \max_i x(i)\right]. (This means that thresholds near the median are less likely to be picked than thresholds far from the median.) This choice seems to be a magic trick in itself, voiding the point I made above, but I hope it will become natural as we unfold our argument.

First, we shall see that there are continuous relaxations of h and \phi that exactly match the combinatorial optimum, and that this can be established via random rounding.

Our first claim is that

(4) \ \ \ \phi(G) = \min_{x \in {\mathbb R}^n} \frac{ \sum_{ij} A(i,j) | x(i)-x(j) |}{\frac 1n \sum_{ij} | x(i)-x(j) |}

To prove (4), take an optimal solution x\in {\mathbb R}^n, pick a random threshold t in the interval \left[\min_i x(i),\max_i x(i)\right], and define S:= \{ i: x(i) \geq t\}. Then it should be clear that the probability that an edge (i,j) is cut by (S,V-S) is proportional to |x(i)-x(j)|. Normalizing x so that \max_i x_i - \min_i x_i = 1, which does not affect the distribution of S, nor the optimality of x, the probability is exactly equal to |x(i)-x(j)|, and we have

{\mathbb E} [edges (S,V-S)] = \frac 12 \sum_{ij} A(i,j) | x(i)-x(j) |

and

{\mathbb E} [|S| \cdot |V-S| ] = \frac 12 \sum_{ij} | x(i)-x(j) |

so that the support of our distribution includes a cut (S,V-S) such that

\frac{edges(S,V-S)} {\frac 1n |S| \cdot |V-S|} \leq \frac{ \sum_{ij} A(i,j) | x(i)-x(j) |}{\frac 1n \sum_{ij} | x(i)-x(j) |}

Next, we want to argue that

(5) \ \ \ h(G) = \min_{x \in {\mathbb R}^n,\ med(x)=0} \frac{ \sum_{ij} A(i,j) | x(i)-x(j) |}{2\sum_i |x(i)|}

where med(x) denotes the median of the values \{ x(1),\ldots,x(n)\}. It may take a minute to see that the right-hand side is a relaxation of h(G). Suppose that (S,V-S) is the optimal cut in the definition of h, and that |S| \leq n/2. Then define x(i)=1 if i\in S and x(i)=0 otherwise. The median of x is zero, |S| = \sum_i |x(i)|, and \sum_{ij} A(i,j) | x(i)-x(j) | counts twice every edge crossing the cut. So the right-hand-side is a relaxation, and can be no larger than h(G).

As before, normalize x so that \max_i x_i - \min_i x_i = 1, pick a threshold t uniformly at random in this interval, and define S = \{ i: x(i) \geq t\}. We already argued that

{\mathbb E} [ edges (S,V-S) ] = \frac 12 \sum_{ij} A(i,j) | x(i)-x(j)|

and it remains to show that

{\mathbb E} [\min \{ |S| , \ |V-S| \} ] = \sum_i |x_i| .

This is just a matter of writing things down. If we name the vertices so that x(1)\leq x(2) \leq \cdots x(n), and we use that for a non-negative integral random variable N we have {\mathbb E} [N] = \sum_k {\mathbb P} [N\geq k], then we just have to observe that

{\mathbb P} [\min \{ |S| , \ |V-S| \} \geq k] = | x(n-k) - x(k)|= |x(n-k)| + |x(k)|.

Now that we have (4) and (5), if someone gives us an x \in {\mathbb R}^n such that

(6) \ \ \ \frac { \sum_{i,j} A(i,j) \cdot | x(i) - x(j)|^2 }{ \frac 1n \sum_{ij} |x(i)-x(j)|^2} = \epsilon

our first thought would be to plug x into the right-hand side of (4) and (5) and hope for the best. Unfortunately, I suspect that such hope would be unwarranted. We can see, however, that

\sum_{i,j} A(i,j) \cdot | x(i) - x(j)|
\leq \sqrt{\sum_{i,j} A(i,j) \cdot | x(i) - x(j)|^2 } \cdot \sqrt{\sum_{i,j} A(i,j)}
= \sqrt{nd \cdot \sum_{i,j} A(i,j) \cdot | x(i) - x(j)|^2}

and

\sum_{ij} |x(i) - x(j)| \geq \frac 1 {\max_{ij} |x(i)-x(j)|} \cdot \sum_{ij} |x(i) - x(j)|^2

so, if (6) holds, we have

\frac{\sum_{ij} A(i,j) |x(i) - x(j)|}{\frac 1n \sum_{ij} |x(i)-x(j)|} \leq \sqrt{\epsilon d} \cdot \frac{\max_{ij} |x(i)-x(j)|}{\sqrt{ \frac 1 {n^2} \sum_{ij} |x(i)-x(j)|}}

and, at least in the special case in which \sum_{ij} |x(i)-x(j)| = \Omega(n^2 \cdot \max_{ij}|x(i)-x(j)|), this gives us a cut of expansion O(\sqrt{\epsilon d}). I don’t think, however, that just using x would work in the general case.

(Note that this special case in which the very simple rounding works is also the case in which the simple rounding gives, on average, a nearly balanced partition. Interestingly, both the Leighton-Rao and the Arora-Rao-Vazirani analyses reduce the general case to a case in which the respective rounding algorithms produce nearly balanced partitions.)

The next try would be to find a y such that |y(i)-y(j)| is an approximation to |x(i)-x(j)|^2, and |x(i)|^2 is an approximation to |y(i)|. My first guess was to choose y(i) := |x(i)|\cdot x(i), and this works!

Let us see what the right-hand sides of (4) and (5) look like for such a solution y. First, we may assume without loss of generality that we are given an x such that the median of x is zero. (Otherwise we can subtract the median from every entry without changing (6).) This means that we have a solution y that is feasible for both (4) and (5).

In order to study the numerator of (6), we need to find a way to relate |x(i)-x(j)| to |y(i)-y(j)|. A couple of attempts give

|y(i) - y(j)| = |x(i)-x(j)| \cdot ( |x(i)| + |x(j)|)

and it’s clear that the next step has to be Cauchy-Schwarz

\quad \sum_{ij} A(i,j) \cdot |y(i) - y(j)|
\leq \sqrt{ \sum_{ij} A(i,j) |x(i)-x(j)|^2} \cdot \sqrt{ \sum_{ij} A(i,j) ( |x(i)| + |x(j)|)^2}

So we have related the numerator of (6) to the numerators of (4) and (5), up to the extra term that is easy to bound (recall the graph is d-regular) as

\ \ \sum_{ij} A(i,j) ( |x(i)| + |x(j)|)^2
\leq \sum_{ij} A(i,j) ( 2\cdot |x(i)| + 2\cdot |x(j)|)
= 4 d \sum_i |x(i)|^2
= 4d \sum_i |y(i)|

Now we use (6) to get

\ \ \ \sum_{ij} A(i,j) |x(i)-x(j)|^2
\leq \epsilon \frac 1n \sum_{i,j} |x(i)-x(j)|^2
= 2 \epsilon \sum_i x_i^2 - 2 \epsilon \left( \sum_i x(i) \right)^2
\leq 2\epsilon \sum_i |y(i)|

and, combining everything,

\frac{\sum_{ij} A(i,j) |y(i)-y(j)|} {2\sum_i |y(i)|} \leq \sqrt{2d\epsilon}.

Although it is unnecessary to complete the proof, we can also see that, because the median of y is zero, we have

\sum_{i,j} |y(i)-y(j)| \geq n \sum_i |y(i)|

and so

\frac{\sum_{ij} A(i,j) |y(i)-y(j)|} {\frac 1n \sum_{ij} |y(i)-y(j)|} \leq 2\cdot \sqrt{2d\epsilon}.

There is another way to look at this “rounding algorithm:” we see the solution x that we are given as defining a “distance function” d(i,j):= |x(i)-x(j)|^2 among the vertices, and we find a combinatorial solution by finding an r such that the cut is defined by the “ball of radius r” around the vertex i_{\min}, where x(i_{\min}) = \min_i x(i). (If t is the threshold picked by the algorithm, then r= |t-x(i_{\min})|.) Indeed, we pick a radius at random, with a certain distribution, and we are able to relate the probability that an edge i,j is cut with the value d(i,j). This view is not very natural for the spectral partitioning algorithm, but it is in the spirit of what happens in the rounding of the Leighton-Rao and the Arora-Rao-Vazirani relaxations.

]]> http://lucatrevisan.wordpress.com/2008/05/11/the-spectral-partitioning-algorithm/feed/ SFIFF 51 http://lucatrevisan.wordpress.com/2008/05/10/sfiff-51/ http://lucatrevisan.wordpress.com/2008/05/10/sfiff-51/#comments Sat, 10 May 2008 18:54:12 +0000 luca http://lucatrevisan.wordpress.com/?p=399

I eventually found my way through the intimidatingly large program of the San Francisco International Film Festival, and to the festival itself.

The movie I most enjoyed was Sleep Dealer. It is set in a near-future where the kind of low-paying jobs now often held by immigrants from developing countries (driving taxis, constructions work, waiting tables) are done by machines that are controlled remotely via a virtual reality technology. And the machines are operated, of course, by low-paid workers in developing countries, so that the richer countries can have all the benefits of immigration, without the immigrants. During the Q&A, the filmmaker Alex Rivera made the point that (as far as he knows) this is the first movie to look into the future of developing countries. One often sees SF movies that imagine the future of New York, or London, but not of Tijuana or Delhi. One woman asked him how he reconciles the political message of his movie with the corporate sponsorship. (She noticed an acknowledgment to Coca Cola in the closing credits.) I liked his response: “we are all steeped in inescapable horror,” he started, “the Gap clothes I am wearing were made in a sweatshop, and the meat I ate for dinner came from animals that were treated inhumanely.” And he went on to say that we cannot fight everything, we have to keep on living, but for the big things, then it is worth trying to push back as much as possible. (I completely agree.)

Big Man Japan, about a middle-aged Japanese super-hero was also a lot of fun.

I also saw two movies from Chinese “sixth generation” directors. Still Life won the golden lion (the top prize) at the 2006 Venice film festival, and I should have known better than to go see it: European film festival juries are populated by the worst kind of film snobs, and watchable movies are not their thing. In the spirit of Italian neo-realismo, the movie is interested in seeing great societal change through the eyes of the “little people” that are affected by them, and via small, disjointed, stories. I don’t get it. I concede, however, that the scenes about the towns being demolished in preparation for the rising level of water after the Three Gorges Dam is completed, are incomparably more moving, if considerably less polished, than the same images in Manufactured Landscapes Umbrella was a fascinating documentary shot as part of a large project that will produce ten documentaries a year for ten years. Following umbrellas from the places where they are produced, to the places where they are traded, to the places where they are used, the movie looks into five classes of Chinese society, factory workers, merchants, college students, soldiers, and farmers. The shots inside a PLA training facility are fascinating, and the final segment was very moving, with a peasant complaining about the rising costs of farming, the lack of welfare, and then reminiscing about the various farming policies from the 1950s on, and finally weeping by just thinking about the time of the cultural revolution.

]]> http://lucatrevisan.wordpress.com/2008/05/10/sfiff-51/feed/ Relaxations of Edge Expansion and Their Duals (Part 2) http://lucatrevisan.wordpress.com/2008/05/05/relaxations-of-edge-expansion-and-their-duals-part-2/ http://lucatrevisan.wordpress.com/2008/05/05/relaxations-of-edge-expansion-and-their-duals-part-2/#comments Tue, 06 May 2008 04:38:44 +0000 luca http://lucatrevisan.wordpress.com/?p=398

[In which we finally get to Leighton-Rao and ARV.]

Continuing our edge expansion marathon, we are going to see more ways in which the sparsest cut problem can be relaxed to polynomial-time solvable continuous problems. As before, our goal is to understand how this implies certificates of expansion.

We are talking about the sparsest cut \phi(G) of a graph G(V,E), which is

\phi(G) := \min_{S\subseteq V} \frac{edges(S,V-S)}{\frac 1n |S|\cdot V-S|} = \min_{x\in \{0,1\}^n} \frac { \sum_{i,j} A(i,j) |x(i)-x(j)|} { \frac 1n \sum_{i,j} |x(i)-x(j)|}

We have looked for a while at the eigenvalues gap of G. If G is d-regular and the second largest eigenvalue of G is \lambda_2, then the difference between largest and second largest eigenvalue obeys

d-\lambda_2 = \min_{x\in {\mathbb R}^n} \frac { \sum_{i,j} A(i,j) |x(i)-x(j)|^2} {\frac 1n \sum_{i,j} |x(i)-x(j)|^2} = \min_{m,x_1,\ldots,x_n\in {\mathbb R}^m} \frac { \sum_{i,j} A(i,j) ||x_-x_j||^2} {\frac 1n \sum_{i,j} ||x_i-x_j||^2}

and, noting that |x(i)-x(j)|^2 = |x(i)-x(j)| when x\in \{0,1\}^n, we saw that d-\lambda_2 is a relaxation of \phi, and \phi \geq d-\lambda_2.

Leighton and Rao derive a continuous relaxation of \phi in quite a different way. They note that when x\in \{ 0,1\}^n, then the “distance”

d(i,j) := |x(i)-x(j)|

induces a semi-metric on V, that is, d(i,i)=0, d(i,j) = d(j,i), and the triangle inequality holds

d(i,j) \leq d(i,k) + d(k,j) \ \ \ \forall i,j,k

One can then consider the relaxation

(1) \ \ \ min_d \frac { \sum_{i,j} A(i,j) d(i,j)} {\frac 1n \sum_{i,j} d(i,j)}

where the minimum is taken over all distance functions d: V\times V \rightarrow {\mathbb R} that define a semi-metric on V.

We can rewrite (1) as the linear program

(2) \ \ \ \begin{array}{l} \min \sum_{i,j} A(i,j) d(i,j)\\ subject\ to \\ \sum_{i,j} d(i,j)=n\\ d(i,j) \leq d(i,k) + d(k,j) \ \ \ \forall i,j,k\\ d(i,j) \geq 0 \ \ \ \forall i,j \end{array}

where we only write the triangle inequality constraints by having one variable d(i,j) for every unordered pair i,j.

The minimum of (2) gives a lower bound to the sparsest cut of G, and so every feasible solution for the dual of (2) gives a lower bound to \phi(G).

Before trying to write down the dual of (2), it is convenient to write (2) in a less efficient way. (This is akin to the fact that sometimes, when using induction, it is easier to prove a more general statement.) Instead of writing the triangle inequality for every three vertices, we write it for every arbitrary sequence of vertices.

(3) \ \ \ \begin{array}{l} \min \sum_{i,j} A(i,j) d(i,j)\\ subject\ to \\ \sum_{i,j} d(i,j)=n\\ d(i,j) \leq \sum_{(u,v) \in p} d(u,v) \ \ \ \forall i,j, \forall p \in P_{i,j} \\ d(i,j) \geq 0 \ \ \ \forall i,j\end{array}

where we denote by P_{i,j} the set of “paths” in the complete graph between i and j. (That is, P_{i,j} is the set of all sequences of vertices that start at i and end at j.)

Clearly (3) is the same as (2), in the sense that the extra inequalities present in (3) are implied by the triangle inequalities in (2). The dual of (3), however, is cleaner, and is given below

(4) \ \ \ \begin{array}{l} \max n\cdot y\\ \ y - \sum_{p\in P_{i,j}} y_p + \sum_{p: (i,j) \in p} y_p \leq A(i,j)\ \ \ \forall i,j\\ y,y_p \geq 0\end{array}

A feasible solution to (4) is thus a weighted set of paths and a parameter y. Before reasoning about the meaning of a solution, suppose that y,y_p satisfy the stronger constraints

(5) \ \ \ \begin{array}{l} \sum_{p\in P_{i,j}} y_p \geq y\\\sum_{p: (i,j) \in p} y_p \leq A(i,j) \end{array}

Then we can view the y_p : p \in P_{i,j} as defining a flow between i,j that carries at least y units of flow; furthermore, the union of all such flows uses at most a total capacity A(i,j) on each edge (i,j). We should think of it as an embedding of the complete graph scaled by y into G. I want to claim that this means that the sparsest cut of G is at least y times the sparsest cut of the complete graph, that is, at least y\cdot n.

To verify the claim, take any cut S,V-S. We know that such a cut separates |S| \cdot |V-S| pairs of vertices i,j, and that at least y units of flow are routed between each such pair, so that at least y\cdot |S| \cdot |V-S| units of flow cross the cut. Since every edge has capacity 1, we see that at least y\cdot |S|\cdot |V-S| edges of G must cross the cut.

It remains to bridge the difference between (4) or (5). One possibility is to show that a solution to (4) can be modified to a solution satisfying (5) with no loss in y. Another approach is to directly use (4) and a general weak duality argument to show directly that a solution satisfying (4) implies that G has at least the edge expansion of a complete graph with edge weights y.

There is, in fact, a broader principle at work, which we can verify by a similar argument: if \{ y_p\} is an assignment of weights to all possible paths (which we think of as defining a multicommodity flow), G is a graph with adjacency matrix A, H is a graph with adjacency matrix B, and we have

B(i,j) - \sum_{p\in P_{i,j}} y_p + \sum_{p: (i,j) \in p} y_p \leq A(i,j)\ \ \ \forall i,j

then we must have \phi(G) \geq \phi(H).

The fact that a feasible solution to (4) gives a lower bound to \phi(G) is a special case that applies to the case in which H is a scaled version of the complete graph.

This is a good point to pause, look back again at the bound coming from spectral considerations, and compare it with what we have here.

(A couple of notes. (i) From now on it will be convenient to identify a graph with its adjacency matrix. (ii) About notation: recall that if G is a d-regular graph then its Laplacian is L(G):= dI - G; more generally, if M is a matrix then its Laplacian L(M) is defined so that L(M)(i,j) := -M(i,j) when i \neq j, and L(M)(i,i) := \sum_j M(i,j). Observe that L is linear, that is, L(A+B) = L(A)+L(B) and L(c\cdot A) = c\cdot L(A).)

In the spectral bound, we find a y such that L(G) - y L(K_n) is positive semidefinite, and we use this fact to prove that \phi (G) \geq y\cdot n. This works because if G and H are two symmetric matrices such that L(G) \succeq L(H), then \phi(G) \geq \phi(H). In the analysis of the spectral gap, we apply it to the case in which H is a scaled version of the complete graph.

An important note: H is not a 0/1 matrix, and in fact it may have negative entries. The quantity \phi(H) is, however, always well defined as

\min_{S\subseteq V} \frac{\sum_{i\in S,j\not\in S} H(i,j)}{\frac 1n |S| \cdot |V-S|}

To wrap up our comparisons, in both cases we prove a lower bound on \phi(G) by showing that \phi(G) \geq \phi(H) where H is a scaled version of the complete graph, so that we know the exact value of \phi(H). In the spectral analysis, we argue that \phi(G) \geq \phi(H) by showing that

L(G) \succeq L(H);

in the Leighton-Rao approach we find a system of multicommodity flows \{ y_p \} such that

G(i,j) \geq H(i,j) + Y(i,j)\ \ \ \forall i,j

where, to clean up the notation, we define

Y(i,j) := \sum_{p: (i,j) \in p} y_p - \sum_{p\in P_{i,j}} y_p.

The two approaches are incomparable, and, naturally, we would like to combine them into a single approach that subsumes both.

Suppose that we find a graph H and flows \{ y_p\} such that

L(G) \succeq L(H+Y)

where, again, Y(i,j) := \sum_{p: (i,j) \in p} y_p - \sum_{p\in P_{i,j}} y_p.

Then, by spectral considerations,

\phi(G) \geq \phi (H+Y)

but by Leighton-Rao reasoning

\phi(H+Y) \geq \phi(H).

This approach subsumes the two because if A,B are two matrices such that A(i,j) \geq B(j,j) for every entry, then L(A) \succeq L(B). (This is “obvious” to people who know some linear algebra, but it is actually a great exercise to get the vector solution that proves that L(A)-L(B) = L(A-B) is positive semidefinite.)

Hence the following semidefinite program produces lower bound certificates for the sparsest cut of G. (As before, we shall take H to be a scaled version of the complete graph.)

(6) \ \ \ \begin{array}{l} \max n\cdot y \\ subject\ to\\ L(G) \succeq L(y\cdot K_n+Y)\\Y(i,j) = \sum_{p: (i,j) \in p} y_p - \sum_{p\in P_{i,j}} y_p\\ y,y_p \geq 0\end{array}

To make sense of it, we need to understand Y and L(Y) a bit better. Let I_{i,j} be the matrix that has ones in the (i,j) and (j,i) entry, and zeroes everywhere else; hence I_{i,j} is the adjacency matrix of the graph consisting of the single edge (i,j). Then

Y = \sum_{i,j} \sum_{p: (i,j) \in p} y_p I_{i,j} - \sum_{p\in P_{i,j}} y_p I_{i,j}
\ \ = \sum_{u,v} \sum_{p\in P_{u,v}} y_p \left( - I_{u,v} + \sum_{(i,j) \in p} I_p \right)

that is,

Y = \sum_p y_p M_p

where M_p is the matrix that has a one in each entry (i,j) corresponding to an edge in p, a -1 in the entry (u,v), where u and v are the end-points of the path p, and zeroes everywhere else. (Equivalently, for a path p, the matrix M_p is the adjacency matrix of the graph that has the path p plus an edge of weight -1 connecting the endpoints of the path.)

Our semidefinite program (6) now takes the form

(7) \ \ \ \begin{array}{l} \max n\cdot y \\ subject\ to\\ y\cdot L(K_n) + \sum_p y_p L(M_p) \preceq L(G)\\ y,y_p \geq 0\end{array}

We again look up in wikipedia how to form the dual of (7), and we get

(8)\ \ \ \begin{array}{l} \min L(G) \bullet X\\subject\ to\\ L(K_n) \bullet X \geq n\\ L(M_p) \bullet X \geq 0 \ \ \ \forall p\\ X \succeq 0 \end{array}

Which is a more constrained version of the semidefinite program that gives the spectral bound. Let us unfold what (8) means, and see that it is indeed a natural semidefinite programming relaxation of \phi, and it subsumes the spectral and the Leighton-Rao relaxations. (Although we already know that this is going to be the case given duality considerations.)

If we let x_1,\ldots,x_n be a system of vectors such that X(i,j) = \langle x_i,x_j \rangle, then from our past work on the spectral relaxation, we know how to make sense of the objective function and of the first constraint.

L(G)\bullet X = \frac 12 \sum_{i,j} G(i,j) \cdot || x_i - x_j||^2

and

L(K_n)\bullet X = \frac 12 \sum_{i,j} || x_i - x_j||^2

We leave it to the reader to verify that if p is a path with endpoints u,v, then

L(M_p) \bullet X= \frac 12 \left(- || x_u - x_v||^2 + \sum_{(i,j) \in p} || x_i - x_j ||^2 \right)

and so the constraints L(M_p) \bullet X \geq 0 are checking that the “distance function” d(i,j) := || x_i - x_j ||^2 satisfies the triangle inequality.

We can rewrite (8) in the following form, which makes it clear that is a relaxation of \phi and a strengthening of both the spectral bound and the Leighton-Rao linear program.

(9) \ \ \ \begin{array}{l} \min \sum_{i,j} G(i,j) \cdot || x_i - x_j ||^2\\subject\ to\\ \sum_{i,j} || x_i - x_j ||^2 = n \\ || x_i - x_j ||^2 \leq ||x_i - x_k||^2 + || x_k - x_j||^2 \end{array}

This is the Arora-Rao-Vazirani semidefinite programming relaxation of sparsest cut.

Notice that if we have a solution x \in \{0,1\}^n to the sparsest cut problem we can turn into a feasible solution to (9) by letting x_i := x(i) be a one-dimensional solution; if we have a feasible solution to (9), then it is feasible for the spectral relaxation (which does not require the triangle inequalities) and it is feasible for Leighton-Rao, by setting d(i,j) := || x_i - x_j ||^2.

And so we have defined our three relaxations of sparsest cut and their duals, and understood “why” dual solutions give certificates of expansion.

It remains to see how close the optima of the relaxations are to \phi, a question that is answered by algorithmic techniques to “round” a relaxed solution into a cut of comparable cost. If we wonder how good such roundings can be, we are led to the study of metric embeddings.

]]> http://lucatrevisan.wordpress.com/2008/05/05/relaxations-of-edge-expansion-and-their-duals-part-2/feed/ Relaxations of Edge Expansion and Their Duals http://lucatrevisan.wordpress.com/2008/05/03/relaxations-of-edge-expansion-and-their-duals/ http://lucatrevisan.wordpress.com/2008/05/03/relaxations-of-edge-expansion-and-their-duals/#comments Sun, 04 May 2008 03:47:17 +0000 luca http://lucatrevisan.wordpress.com/?p=397

In the previous post, we saw that the sparsest cut of a graph G=(V,E) (which is within a factor of 2 of the edge expansion of the graph), is defined as

\phi := \min_{S\subseteq V} \frac{edges(S,V-S)}{\frac 1n |S|\cdot |V-S|}

and can also be written as

(1) \ \ \ \phi = \min_{x\in \{0,1\}^V} \frac{\sum_{i,j} A(i,j) | x(i)-x(j)|}{\frac 1n \sum_{i,j} |x(i)-x(j)|}

while the eigenvalue gap of G, can be written as

(2) \ \ \ d-\lambda_2 = \min_{x\in {\mathbb R}^V} \frac{\sum_{i,j} A(i,j) | x(i)-x(j)|^2}{\frac 1n \sum_{i,j} |x(i)-x(j)|^2}

and so it can be seen as a relaxation of \phi, considering that
|x(i)-x(j)|=|x(i)-x(j)|^2 when x(i),x(j) are boolean.

Let us now take a broader look at efficiently computable continuous relaxations of sparsest cut, and see that the eigenvalue gap is a semidefinite programming relaxation of sparsest cut, that by adding the triangle inequality to it we derive the Arora-Rao-Vazirani relaxation, and that by removing the semidefinite constraint from the ARV relaxation we get the Leighton-Rao relaxation.

We first prove that (2) is equivalent to

(3) \ \ \ d-\lambda_2 = \min_{m, x_1,\ldots ,x_n \in {\mathbb R}^m} \frac{\sum_{i,j} A(i,j) || x_i - x_j ||^2}{\frac 1n \sum_{i,j} || x_i -x_j ||^2}

The difference is that instead of optimizing over all possible ways to assign a real number x(i) to each vertex i, we are now optimizing over all possible ways to assign a vector x_i to each vertex i. The absolute value |x(i)-x(j)| is then replaced by the Euclidean distance || x_i - x_j ||. The right-hand-side of (3) is clearly a relaxation of the right-hand-side of (2), so we just need to establish that, in the right-hand-side of (3), it is optimal to take 1-dimensional vectors. For every vector solution x_1,\ldots,x_n, we see that

\frac{\sum_{i,j} A(i,j) || x_i - x_j ||^2}{\frac 1n \sum_{i,j} || x_i -x_j ||^2} = \frac{\sum_k \sum_{i,j} A(i,j) | x_i(k) - x_j(k) |^2}{\sum_k \sum_{i,j} | x_i(k) -x_j(k) |^2} \geq \min_k \frac{\sum_{i,j} A(i,j) | x_i(k) - x_j(k) |^2}{\sum_{i,j} | x_i(k) -x_j(k) |^2}

where the last inequality follows from

(4) \ \ \ \frac{\sum_{i=1}^n a_i}{\sum_{i=1}^n b_i} \geq \min_i \frac {a_i}{b_i}

which, in turn, has a one-line proof:

\sum_i a_i = \sum_i b_i \cdot \frac {a_i}{b_i} \geq \left( min_i \frac {a_i}{b_i} \right) \cdot \sum_i b_i.

The formulation (3) is equivalent to the following semidefinite program:

(5) \ \ \ \begin{array}{l} \min \sum_{i,j} A(i,j) ||x_i -x_j ||^2\\ subject\ to\\\sum_{i,j} ||x_i -x_j ||^2 = n\\x_i \in {\mathbb R}^{*} \end{array}

An optimization problem is a semidefinite program if every variable is allowed to be an arbitrary vector, and both the objective function and the constraints are linear functions of inner products of pairs of variables. In our case,

||x_i - x_j||^2 = \langle x_i , x_i \rangle + \langle x_j , x_j \rangle - 2 \langle x_i , x_j \rangle

is a linear combination of inner products, and so the formulation (5) is a semidefinite program. Another way to define semidefinite programming (indeed, the definition that is usually given first, deriving the one in terms of inner products as an equivalent characterization) is that a semidefinite program is an optimization problem in which the variables are the entries of a positive semidefinite matrix X, and the objective function and the constraints are linear in the entries of X. A real matrix is positive semidefinite if it is symmetric and all its eigenvalues are non-negative. We can see that an n\times n matrix X is positive semidefinite if and only if there are vectors x_1,\ldots,x_n such that X(i,j) = \langle x_i,x_j \rangle.

(The proof is simple: if X is real and symmetric, then all its eigenvalues \lambda_1,\ldots,\lambda_n are real, and there is an orthonormal system of eigenvectors v_1,\ldots,v_k such that

X=\sum_k \lambda_k v_k \cdot v_k^T

so that

X(i,j) = \sum_k \lambda_k v_i (k) v_j(k)

If the eigenvalues are non-negative, we can let x_i(k) := \sqrt{\lambda_k} v_i(k) and have X(i,j) = \langle x_i ,x_j\rangle.

On the other hand, if there are x_i such that X(i,j) = \langle x_i,x_j\rangle, let \lambda be any eigenvalue of X and v be a length-1 eigenvector of \lambda. Then:

\lambda = v^T X v = \sum_{i,j} v_n(i)v_n(j) \sum_k x_i(k) x_j(k) = \sum_k \left( \sum_i v_n(i) x_i(k) \right)^2 \geq 0

which completes the characterization.)

So we have gotten to the point where we can see d-\lambda_2 as a semidefinite programming relaxation of sparsest cut. An interesting property of semidefinite programs is that, like linear programs, they admit a dual. From a minimization (primal) semidefinite program, we can write a maximization (dual) semidefinite program such that the cost of any feasible solution for the dual is a lower bound to the cost of any feasible solution for the primal; furthermore, the two programs have the same optimum cost. Returning to our motivating question of certifying the expansion of a given graph, any feasible solution to the dual of (5) gives us a lower bound to the expansion of the graph.

We shall have to suffer a bit longer before we can write the dual of (5). It is best to first move to a formulation in terms of entries of semidefinite matrices. A few calculations show that (5) can be written as

(6) \ \ \ \begin{array}{l} \min L_G \bullet X\\ subject\ to\\L_n \bullet X =n\\ X \succeq 0 \end{array}

Where we write X \succeq 0 to mean that the matrix X is positive semidefinite (we shall also write X\succeq Y when X-Y is positive semidefinite), we write A \bullet B to mean \sum_{i,j} A(i,j)B(i,j), L_G is the Laplacian of G, that is the matrix dI - A, and L_n is the Laplacian of the clique on n vertices.

From this, we look up in Wikipedia the rules to form a semidefinite dual, and we get

(7) \ \ \ \begin{array}{l} \max y\\ subject\ to\\L_G \succeq y\frac 1n L_n \end{array}

It is possible to make sense of (7) independently of all the above discussion, that is, to see from first principles that the cost of any feasible solution to (7) is a lower bound to the sparsest cut of G.

A feasible solution is a value y and a positive semidefinite matrix Y such that L_G = y\frac 1n L_n + Y. Equivalently, it is a value y and vectors y_i such that

L_G(i,j) = y\frac 1n L_n (i,j) + \langle y_i,y_j\rangle

Take any cut (S,V-S) in G. Then

|S|\cdot |V-S| = 1_S^T L_n 1_S

and

edges(S,V-S) = 1_S^T L_G 1_S

but

1_S^T L_G 1_S = 1_S^T (y \frac 1n L_n + Y) 1_S \geq 1_S^T y \frac 1n L_n 1_S

and so the cost of the cut is at least y. The justify the last inequality above, we need to prove that if Y is positive semidefinite than

1_S^T Y 1_S \geq 0

Indeed

1_S^T Y 1_S = \sum_{i,j\in S} \langle y_i,y_j \rangle = \langle \sum_{i\in S} y_i,\sum_{i\in S} y_i \rangle = || \sum_{i\in S} y_i ||^2

We can also see, from first principles, that if \lambda_1=d,\lambda_2,\ldots,\lambda_n are the eigenvalues of A and v_1,\ldots,v_n is a corresponding system of eigenvectors, we can find a feasible solution to (7) of cost d-\lambda_2.

We defined L_G = dI - A, where we assume that G is d-regular. We can write

dI - A
= dI - \sum_i \lambda_i v_i v_i^T
= (d-\lambda_2) \cdot\left( I - v_1v_1^T \right) +\lambda_2 I - \left( \lambda_2 v_1v_1^T+ \sum_{i=2}^n \lambda_i v_i v_i^T \right)

And we are almost there because \left( I - v_1v_1^T \right) = \frac 1n L_n, and it remains to prove that the matrix

Y:= \lambda_2 I - \left( \lambda_2 v_1v_1^T+ \sum_{i=2}^n \lambda_i v_i v_i^T \right)

is positive semidefinite. This we can see by noting that v_1,\ldots,v_n are eigenvectors of Y with eigenvalues 0,0,\lambda_2-\lambda_3,\ldots,\lambda_2-\lambda_n. Alternatively, we could construct vectors y_i such that Y(i,j) = \langle y_i,y_j\rangle.

Essentially, what is going on is that we are “embedding” a scaled version of the complete graph into our graph. The edge expansion and sparsest cut of the complete graph are known, and the semidefinite constraint in (7) is telling us that the difference between our graph and the scaled complete graph can only add to the sparsest cut, so the sparsest cut of G is at least the known sparsest cut of the scaled complete graph.

Why did we go through all this trouble? Now that we understand the spectral gap inside out, we can derive the Arora-Rao-Vazirani relaxation (ARV) of sparsest cut by just adding the “triangle inequalities” to (5), and we can derive the Leighton-Rao relaxation by removing the semidefinite constraints from ARV. In the dual of Leighton-Rao, we again “embed” a complete graph in our graph, by realizing every edge as a flow. In the dual of ARV, we will find both flows and spectral considerations.

[to be continued]

]]> http://lucatrevisan.wordpress.com/2008/05/03/relaxations-of-edge-expansion-and-their-duals/feed/ The Spectral Lower Bound to Edge Expansion http://lucatrevisan.wordpress.com/2008/04/30/the-spectral-lower-bound-to-edge-expansion/ http://lucatrevisan.wordpress.com/2008/04/30/the-spectral-lower-bound-to-edge-expansion/#comments Wed, 30 Apr 2008 23:45:28 +0000 luca http://lucatrevisan.wordpress.com/?p=396

In which we dwell at great length on the “easy” part of Cheeger’s inequality.

The edge expansion of a graph G=(V,E), defined as

h(G):= \min_{S: |S| \leq \frac {|V|}{2} } \frac{edges(S,V-S)}{|S|}

is a measure of how “well connected” is a graph. A graph has edge expansion at least h if and only if, in order to disconnect a set of k vertices from the rest of the graph, one must remove at least hk edges. In other words, the removal of t edges from a graph of edge expansion \geq h must leave a connected component of size \geq n - t/h, where n is the number of vertices.

As readers of in theory know, graphs of large edge expansion (expander graphs) have many applications, and in this post we are going to return to the question of how one certifies that a given graph has large edge expansion.

For simplicity, we shall only talk about regular graphs. In a previous post we mentioned that if A is the adjacency matrix of a graph, there are numbers \lambda_1 \geq \lambda_2 \geq \cdots \geq \lambda_n (the eigenvalues of A), not necessarily distinct, and an orthonormal basis x_1,\cdots,x_n of {\mathbf R}^n, such that for all i

x_iA = \lambda_i x_i

And if the graph is d-regular, then \lambda_1=d, x_1 = \frac {1}{\sqrt n} (1,\cdots,1), and for all i we have |\lambda_i | \leq d. Finally, the numbers \lambda_i and the vectors x_i are computable in polynomial time. (This is all the linear algebra we will need.)

Given this setup, we have the lower bound

h(G) \geq \frac 12 \cdot (d-\lambda_2) \ \ \ (1)

That is, the difference between largest and second largest eigenvalue gives a lower bound to the edge expansion of the graph.

Although (1) is very easy to prove, it is a remarkable result, in the sense that it gives a polynomial-time computable certificate of a “coNP” statement, namely, that for all cuts, at least a certain number of edges cross the cut. Since computing edge expansion is NP-complete, the inequality cannot always be tight. Eventually shall compare it with other methods to lower bound the edge expansion, and see that they are all special cases of the same general approach. To begin with, in this post we shall show that d-\lambda_2 is an efficiently computable relaxation of the edge expansion problem. (Or, rather, of the closely related sparsest cut problem.)

We define the sparsest cut of a graph as

\phi(G):= \min_{S} \frac{edges(S,V-S)} {\frac {1}{n} \cdot |S| \cdot |V-S|}

Clearly we have

\phi \geq h \geq \frac 12 \cdot \phi

so we need to prove \phi \geq d-\lambda_2. We will do so by showing that d-\lambda_2 is a relaxation of \phi, that is, the computation of d-\lambda_2 can be seen as an optimization problem that is identical to \phi, except that we optimize over a bigger range of feasible solutions. To get started, we need to think about eigenvalues as solutions to optimization problems. This comes from characterizing eigenvalues via Rayleigh quotients. For our purposes (in which A is the adjacency matrix of a d-regular graph) we shall use the following simple identity

\lambda_2 = \max_{x: x \perp (1,\cdots 1)} \frac {x^TAx} {x^Tx}\ \ \ (2)

To prove the identity, take any vector x orthogonal to (1,\cdots,1) (and hence to