An Alternative to the Seddighin-Hajiaghayi Ranking Methodology

[Update 10/24/14: there was a bug in the code I wrote yesterday night, apologies to the colleagues at Rutgers!]

[Update 10/24/14: a reaction to the authoritative study of MIT and the University of Maryland. Also, coincidentally, today Scott Adams comes down against reputation-based rankings]

Saeed Seddighin and MohammadTaghi Hajiaghayi have proposed a ranking methodology for theory groups based on the following desiderata: (1) the ranking should be objective, and based only on quantitative information and (2) the ranking should be transparent, and the methodology openly revealed.

Inspired by their work, I propose an alternative methodology that meets both criteria, but has some additional advantages, including having an easier implementation. Based on the same Brown University dataset, I count, for each theory group, the total number of letters in the name of each faculty member.

Here are the results (apologies for the poor formatting):

1 ( 201 ) Massachusetts Institute of Technology
2 ( 179 ) Georgia Institute of Technology
3 ( 146 ) Rutgers – State University of New Jersey – New Brunswick
4 ( 142 ) University of Illinois at Urbana-Champaign
5 ( 141 ) Princeton University
6 ( 139 ) Duke University
7 ( 128 ) Carnegie Mellon University
8 ( 126 ) University of Texas – Austin
9 ( 115 ) University of Maryland – College Park
10 ( 114 ) Texas A&M University
11 ( 111 ) Northwestern University
12 ( 110 ) Stanford University
13 ( 108 ) Columbia University
14 ( 106 ) University of Wisconsin – Madison
15 ( 105 ) University of Massachusetts – Amherst
16 ( 105 ) University of California – San Diego
17 ( 98 ) University of California – Irvine
18 ( 94 ) New York University
19 ( 94 ) State University of New York – Stony Brook
20 ( 93 ) University of Chicago
21 ( 91 ) Harvard University
22 ( 91 ) Cornell University
23 ( 87 ) University of Southern California
24 ( 87 ) University of Michigan
25 ( 85 ) University of Pennsylvania
26 ( 84 ) University of California – Los Angeles
27 ( 81 ) University of California – Berkeley
28 ( 78 ) Dartmouth College
29 ( 76 ) Purdue University
30 ( 71 ) California Institute of Technology
31 ( 67 ) Ohio State University
32 ( 63 ) Brown University
33 ( 61 ) Yale University
34 ( 54 ) University of Rochester
35 ( 53 ) University of California – Santa Barbara
36 ( 53 ) Johns Hopkins University
37 ( 52 ) University of Minnesota – Twin Cities
38 ( 49 ) Virginia Polytechnic Institute and State University
39 ( 48 ) North Carolina State University
40 ( 47 ) University of Florida
41 ( 45 ) Rensselaer Polytechnic Institute
42 ( 44 ) University of Washington
43 ( 44 ) University of California – Davis
44 ( 44 ) Pennsylvania State University
45 ( 40 ) University of Colorado Boulder
46 ( 38 ) University of Utah
47 ( 36 ) University of North Carolina – Chapel Hill
48 ( 33 ) Boston University
49 ( 31 ) University of Arizona
50 ( 30 ) Rice University
51 ( 14 ) University of Virginia
52 ( 12 ) Arizona State University
53 ( 12 ) University of Pittsburgh

I should acknowledge a couple of limitations of this methodology: (1) the Brown dataset is not current, but I believe that the results would not be substantially different even with current data, (2) it might be reasonable to only count the letters in the last name, or to weigh the letters in the last name by 1 and the letters in the first name by 1/2. If there is sufficient interest, I will post rankings according to these other methodologies.

94 thoughts on “An Alternative to the Seddighin-Hajiaghayi Ranking Methodology

  1. Congratulations, Anand!

    As for us, even Christos’s and Prasad’s strong contributions were not enough to push U.C. Berkeley to the top.

  2. Right, because paper count has absolutely nothing to do with scientific quality You might as well count letters.

    I’m going to go out on a limb and predict that if we limit to the last 7 years only, rank each paper manually and add all contributions with a proper score from 0 to 10, regardless of venue we wouldn’t get rankings much different than those of Hajiaghayi. This follows from basic statistics.

    To wit, the correlation between quality and quantity is generally high as illustrated by the rank 1 vs. rank1 +1/2 rank 2, and references to Goedel or Valiant who are the rare exceptions are red herrings. Surely there is also a strong correlation between life time output and last 7 years output. Similarly we also all believe that as a group better papers tend to appear in better conferences. So to sum up, we use three strongly correlated values to measure the actual signal, with slight errors on any one of those likely cancelling each other out.

    So then why are we getting all upset about the rankings? Simple, as the saying goes “comparisons are odious”, particularly when they attempt to linearly order objects that likely do not form a total order. But this has nothing to do with the quality of the rankings.

    So let’s stop pretending that our displeasure is with the methodology.

    Rankings suck, but so long as they exist Hajiaghayi’s are not in any way particularly insidious.

  3. I do not understand your point: even US news is using some measures for ranking schools e.g. tuition, number of graduate students, average GRE scores, etc (e.g. see http://grad-schools.usnews.rankingsandreviews.com/best-graduate-schools/top-engineering-schools/computer-engineering-rankings?int=982d08) though still not transparent for the weights. US news does not use such measures for say CS theory ranking and its seems very reasonable to suggest some measures.

  4. Luca, with all respect your stupidity in this post does not devalue their work. Very sorry to see such a post in your blog.

  5. Even with my name, Harvard ends up low on the ranking. I therefore suggest refining your scoring mechanism so that only names (first and last) beginning with “M” are counted in the total.

  6. @seriously people: No Hajiyaghayi’s is not particularly insidious, it is just him indulging into something habitually insidious.

    Also, “this follows from basic statistics” is not a rigorous statement dude. You’d know if you work in theory!

  7. You know what would be hilarious: next time in the products/publications section of your NSF biosketches, you and Michael M. just report the number of letters of the titles of your papers.

  8. (How about next time you have to give a prospective graduate student career advice, you tell them to go to UMD instead of Stanford because of Taghi’s ranking)

  9. Luca, your script is an absolutely worthless way to rank departments.

    Everybody knows that an umlaut or diacritic should count as a separate letter (or perhaps you should count only umlauts).

  10. @Zenaphora: Taghi’s ranking accurately picks up the fact that there was a period during the last two decades in which Stanford had few active theory faculty. If we were to limit the paper count to last seven years only, I’m sure it would move up in the rankings a tad, since they’ve hired heavily since those dark days.

  11. brilliant/hilarious. nobody has pointed this out yet but presumably # of letters in names at schools is roughly correlated with # of people working in those departments. which leads to the obvious idea that maybe rankings at each school should be weighted by total # of faculty. ie a school with double the # of faculty can produce double the # of papers.
    alas we can laugh a bit at this but rankings are real, but sometimes hidden/ informal as used on hiring committees etc. an effort to make informal/ hidden/ nontransparent rankings more transparent seems noble in general. the devil is in the details.
    reminds me of Goodhart law, Campbells law, Sayres law, Parkinsons law.

  12. “an effort to make informal/ hidden/ nontransparent rankings more transparent seems noble in general. the devil is in the details.”

    This. We can make silly jokes but rankings are taking place as we speak right now, with little input over them and being used by all manner of people such as bureaucrats, prospective students and faculty members, and award committees.

    We can influence Taghi’s rankings since he’s a colleague and going about it in pretty open ways. And for the record, I think the rankings ought to be improved in various ways such as last 7 years only, accounting for coauthorship and attenuating the size-of-group signal, but I doubt any of this will happen now given the vitriolic response of the community.

  13. The ranking I personally use is: if you are looking for an academic career look at the number of people employed in places you want to work graduate from each university (or supervised by each potential adviser).

  14. You should just count the occurrence of these 6 letters {T, H, E, O, R, Y}. None of other letters matter to theory.

  15. Agreed with seriously people. Making silly jokes from respected researchers and the community as a whole is irresponsible.

  16. @anon 2:02pm

    So just because he released a flawed first attempt he deserves his honesty and publication record questioned and his attempt caricatured?

    You certainly would not write a review like that to a poor paper, then why do you feel it is ok when it comes to an attempt to rank theory groups, flawed as it might have been.

    The people who come out looking good here are Taghi and those who politely, professionally pointed out flaws in the ranking.

    Those who wrote ad hominem attacks under a veil of anonymity on Taghi’s research record and integrity are the ones who should hang their head in shame. So much so that Lance’s had to close his blog to comments and Luca had to delete some from this blog as well.

    But at the end of the day we agree, I too expected a classier, more professional level of discussion.

    p.s. it is interesting how many people immediately dismissed the numbers as delusional, without even examining the data. I bet most people questioning UMD’s ranking couldn’t even tell you the names of the researchers in their theory group, which for the record are:

    Alexander Barg
    William Gasarch
    Mohammad Hajiaghayi
    Joseph Ja’ Ja’
    Jonathan Katz
    Samir Khuller
    Clyde Kruskal
    David Mount
    Elaine Shi
    Aravind Srinivasan
    Uzi Vishkin
    Yan Huang
    Dana Dachman-Soled
    Eunhui Park

    That’s a solid group of theory researchers, if you ask me.

  17. @Seriously people
    If you sincerely believe what you just say, could you please leave a non-anonymous comment and state your identity? If not, mind explaining to us why? After all, you are making comments against people who criticize Taghi under anonymity.

  18. @anon 3:17 writes “after all, you are making comments against people who criticize Taghi under anonymity.”

    I have no issue with people who pointed out flaws on the ranking politely and professionally under the veil of anonymity. In fact, there are good reasons why those criticisms are sometimes best made anonymously, e.g. blind reviews.

    I only called on those “who wrote ad hominem attacks under a veil of anonymity”. I assume we all know the difference between being critical of someone else and an ad-hominem attack.

  19. I do not understand why there should be so much uproar here. Note that SH do not call their ranking CS Theory ranking: they just call it “Ranking of CS Departments based on the Number of Papers in Theoretical Computer
    Science”. Also this blogger may call his ranking “ranking of cs
    departments based on the total number of letters in the name of
    each faculty member”. So then these are real measure and anyone can take it or leave it. Indeed the one who claims has CS
    theory ranking is US news without mentioning any methodology. Then lots of departments blindly and proudly put these rankings in
    their first pages even without mentioning the methodology.

  20. To ee: “We feel there is a lack of transparency and well-defined measures in the methods used by U.S. News to rank CS departments in theoretical computer science (and other similar rankings). Over the past several months we have developed a ranking based on a hard, measurable method for the top 50 U.S. universities.” How does this sound to you? The objective was not so modest as you are suggesting. UMD has very respectable researchers and I am sure they do not endorse this ranking either.

  21. To to ee: come on they say “developed A ranking based on ” and not THE ranking. As far as I know in my life-time career, if somebody wants to claim something they emphasize it on the title. You cannot cherry-pick some sentence and base your reasoning on that.

    BTW I believe you are contradicting yourself in “UMD has very respectable researchers and I am sure they do not endorse this ranking either”. As far as I see the main issue with the ranking raised by some is mainly the place of UMD and maybe UCSD and otherwise the ranking is making much more sense say than US news. So you say UMD has very respectable researchers but they should say we are not good enough and sorry that this is our record of publications. It’s funny:)

  22. Earlier I said I would’ve appreciated something classier from Luca. Upon further thought I changed my mind.

    I now consider his post optimal. Engaging in “professional” dialogue about the Taghi ranking would’ve given it a legitimacy that it does not deserve.

    No amount of dialogue compensates for a crafty beginning.

  23. Funny, but in my opinion Taghi’s ranking are superior to the closed source US News report which once had a department with no theoreticians in the top 10 and another with two theoreticians in the top four. Yet not only do we not see the same scorn, we even had some famous researchers banding them about as a good comparison point.

    Or to use a more recent example which provoked zero response from the community, here are the utterly absurd QS rankings from earlier this year:

    http://www.topuniversities.com/university-rankings/university-subject-rankings/2014/computer-science-information-systems#sorting=rank+region=+country=+faculty=+stars=false+search=

    Again where was the outrage?

  24. I’m really proud to contribute to Rutgers’ high ranking. I knew there was a reason I hyphenated my last name!

  25. The discussion so far surprised me. I thought we are standing up against all kind of rankings. But if we are to follow any ranking, Taghi’s makes way more sense than USNews, QS, Times etc. All schools publish news-stories based on these rankings. Look at those crybabies now that UCSD published one.
    Taghi’s ranking is more straightforward. It counts the number of papers. Luca’s ranking also makes more sense than USNews – it is correlated to the size of the group. UCSD published more papers than Stanford. UMD has a more prolific theory group than Harvard. Oh – the gays and the blacks are now dining with us – what a disaster (now we have to make a point that size does not matter – and seriously, it does not).

  26. Thanks Seriously people for listing UMD theory group (that admittedly I was not aware of them all). I am just looking on their webpages: they have a complete and excellent algorithm group working on all areas of algorithms including randomized algorithms, approximation algorithms, game theory, fixed-parameter, parallel algorithms and also have a strong crypto/security group. This strenght shows exactly on their publication numbers in Taghi’s excel file. What’s the uproar then?

  27. The problem is not necessarily with the specifics of the ranking presented by Taghi. The problem is with the absurd pretence and classles way in which a ranking, which is a kind of necessary evil, is presented to the public as an achievement to celebrate.

    For example, I would also not have liked to hear about a ranking of, say, torture methods; where one presents “a ranking of the methods measuring the optimal amount of information extracted from hostages”. Though I may acknowledge that torture under some circumstances is legitimate. I’m not comparing the morality of rankings to torture, I merely point out that both should be handled with care and sensitivity, that lacked the original announcement of Taghi. Causing much of the mocking pointed at his ranking.

  28. Paul, exactly what kind of care and sensitivity are you expecting from Taghi’s announcement? Perhaps you could enlighten us by showing how you would write the announcement yourself?

  29. Rankings can help prospective grad students. Although this might be their only good, It’s not bad to have one.
    Taghi could have discussed what he plans to do (before doing it) with the community. As the result, he probably would have gotten a ranking with lower rank for Maryland, but higher respect for the ranking itself !

  30. Okay. Let me enlighten you with an imperfect sketch. The idea is to be more reserved and modest.


    The following is a simple methodology for ranking departments using simple counting of papers. Although we believe that ranking departments can never be accurate and should be avoided if necessary, sometimes there are situations where such rankings are necessary, e.g. for grants …

    We do not claim our ranking represents an objective total order rather we aim at giving an example of a simple methodology which is transparent and thus may be considered better in certain circumstances.

    For reasons of fairness we shall not consider UMD as part of the ranking…

  31. Thanks Paul. Well written. Now I wish they would have made the announcement that way – so we won’t need to witness the ugly side of the community, which I find disgusting.

  32. Paul (and anon 10:30), though I understand some of your rewordings,
    but I think your view is a bit simplistic. Don’t they say “Ranking
    of CS Departments based on the Number of Papers in Theoretical
    Computer Science”. Don’t they say “Disclaimer: If you find the
    ranking in this website offensive, please ignore it.”? Don’t they
    say: “Our method for ranking is very simple (to avoid any
    complications by different measures)”, Don’t they say “We gather all comments and possibly errors for this Beta version”, Dont’t they say “We gathered the list of Algorithms and Theory conferences and their ranks from this website” [to be fair], etc. Do you really think
    some people attack because of wording or because they do not want to
    lose their high-spots in US News ranking, which just claims “THE”
    ranking without any explanation, by ANY ranking. Also regarding removing UMD, in retrospect, why don’t you delete UMD for now see the measure and the corresponding ranking (without UMD) and if it makes sense return UMD as well. Surely if they discussed this beforehand, such a community just discouraged them for doing anything.

  33. People criticizing Haghi may have made some classless comments, but people supporting him are basically calling for others to shut up.

    It is worth noting that censorship is far worse that whatever alleged lack of class.

  34. wow, this was a long defensive comment based on weak logic.
    and btw, David, If you find this comment offensive, please ignore it.

  35. David, sleighs of hand cannot change the main message. Suppose I say

    —————-
    Jim is a weak researcher.
    —————-

    This is clearly an attack on Jim as a researcher. Now, suppose I say

    —————-
    Jim is a weak researcher.*

    * If you find this comment offensive, please ignore it.
    —————-

    Does this help? Or, suppose I say

    —————-
    The results of our MATLAB simulation indicate that Jim is a weak researcher.*

    * If you find this comment offensive, please ignore it.
    —————-

    Does this then help?

  36. “but people supporting him are basically calling for others to shut up.”

    No one has said such a thing, which suggests this thread is reaching the end of its useful life.

    The comment from @David is no better. David’s comment is full of relevant quotes and well argued. @David simply calls it “weak logic” and provides no reasoning for it.

    So this too suggests we have reach the end of intelligent discussion—not that we had a very good beginning either.

    tl;dr:

    1) Rankings are a necessary evil. 2) Taghi’s ranking could have been better, though 3) it is not as egregious as the anti-UMD people claim. 4) The tone and nature of some of the criticism was embarrassing.

  37. @Seriously People
    I am not sure everyone would agree to (1) “Rankings are a necessary evil”. For instance, I thought the takeaway point from Luca’s post is that rankings are inherently useless – he just present his argument in a funny manner.

  38. @Seriously people, you yourself wrote:

    —————-
    Those who wrote ad hominem attacks under a veil of anonymity on Taghi’s research record and integrity are the ones who should hang their head in shame. So much so that Lance’s had to close his blog to comments and Luca had to delete some from this blog as well.
    —————-

    This is a straw-man argument (implying that people criticizing Taghi are “writing ad hominem attacks” etc) quickly followed by a call to censorship (“should hang their head in shame”).

  39. “@David simply calls it “weak logic” and provides no reasoning for it.”

    I thought it doesn’t need clarification: I called it “weak logic” just to show that the argument that Taghi (and his student) use is kinda funny, i.e. “If you find it offensive, ignore it”.

  40. @anon 1:48pm

    Good point. Let me rephrase: 1) rankings happen and are used by people whether we like it or not.

  41. Taghi could at least add some flexibility to his code and let people see how the ranking changes if they play with its parameters.

    Announcing a ranking like this, without leaving any place for people’s opinions is closer to making a statement (and maybe advertisement) than a sincere contribution

  42. @David, Taghi original post (on the COMPUTATIONAL COMPLEXITY blog) didn’t have any of the reservations you wrote.

    I understand that these disclaimers appeared elsewhere. But this just goes to show carelessness and a lack of judgement: rankings are extremely sensitive matter, and should be avoided if possible.

  43. @David, anon 10:30 here. I do worry that you are right – but I would like to give benefit of doubts to those folks…

  44. “Taghi could at least add some flexibility to his code and let people see how the ranking changes if they play with its parameters.”

    Some people politely asked and guess what “we plan to release a customizable version like that in the future.” as reported at the bottom of this page:

    http://projects.csail.mit.edu/dnd/ranking/

    So far none of the commercial rankings allow you to do that, not US News, QS, not Times, not Shanghai, not Leiden. QS allows you to rank by a single columns (user choice) but not to reweigh the columns.

    This brings me back to one of my original points. By the time we are done, a few months from now, Taghi’s ranking has a chance of being the best because it is open and controlled by a colleague.

    The commercial rankings on the other hand have stunk for decades and we haven’t been had much success in effecting change in them.

  45. I didn’t think this deserved such a long discussion, but since it seems to have generated one, let me add a few non-anonymous points.

    First, and perhaps most importantly, people reading blogs should not interpret anonymous comments as representing the community in any way. I and many others I know would never comment anonymously. Specifically, while I didn’t read carefully all comments I did see a few distasteful ones (especially in Lance’s blog) with unwarranted personal attacks on MohammadTaghi Hajiaghayi.

    Regarding the ranking itself, I find rankings very useful when looking for a place to grab lunch, but not so useful in making major life decisions such as where to work or study. Thus I think we can do just fine without them, and don’t see what’s so important about producing yet another ranking.

    In particular, I think scientists should push back against attempts to quantify scientific productivity by paper counting, which makes even less sense than measuring programmers productivity by lines of codes, and could create twisted incentives in our publication culture. So, regardless of the actual ranking, I actually prefer the methodolody of US News to this one, since I see less of a danger with people trying to maximize their reputation with department heads and directors of graduate studies than with people trying to maximize paper count. (Just to clarify, my distaste for such quantitative measures of research productivity would stand the same even if this was normalized by co-authors, department size, citations, publication year, etc..)

    There is a tendency to think of automated ranking algorithms as somehow “objective” or “fair” as opposed to decisions made by people, but I think this is a rather naive and inaccurate view. For a more illuminating discussion see this wonderful paper of Dwork et al. Mike Ananny (former postdoc at MSR New England) has also worked on this area of “ethics of algorithms”.

  46. I mostly agree with Boaz, and definitely agree on the fact that one should never think of anonymous comments as being representative. I have a strong dislike with the practice of deleting comments, but I had to do it a few times in this thread.

    But my main point is slightly different from Boaz’s. There are circumstances when people have to pick among competing offers, when they are admitted to a graduate program, when they receive post-doc offers and when they receive faculty offers. In such cases, one should think about what place is best for himself, or for herself, and this will change from person to person, and it cannot be reduced to a linear order. As a general advice, I would agree with what an anonymous (ah!) commenter wrote above:

    “if you are looking for an academic career look at the number of people employed in places you want to work graduate from each university (or supervised by each potential adviser).”

    Of course the fact that I and many others do not like linear rankings will not stop such rankings from existing. So how should we think about them? I see two big problems with all “objective” ranking schemes based on bibliometric data or any kind of data.

    One is that (I am sure this concept has a name) once a measure that correlates with quality is known to be used for ranking, it stops being a good measure for quality because people start gaming it. So if everybody believed that being a good researcher or a good department is measured by how many STOC or FOCS paper one has, this would mess up the way STOC and FOCS work; and if it was decided that some citation-based measure is the one to use, it would mess up the way people cite other work, and so on. That is, such rankings would not be merely useless, they would be actively damaging.

    The other is that, in some sense, reputation-based rankings are “complete,” meaning that if any ranking scheme works, reputation-based ranking works, and one might as well use it. This is where the ranking used in this post comes in. How do you disprove that the method I used in my ranking is not good? Well, it puts UC Berkeley in 27th place, and it puts the Rensselaer Polytechnic Institute ahead of the University Washington. But why is it a sign that the rankings are bad? Because everybody knows that the University of Washington is a better place for theory than the Rensselaer Polytechnic Institute, and that UC Berkeley is, let’s say, one of the top 10 places for theory.

    In fact, whenever one wants to criticize a ranking scheme, one always compares it with reputation-based rankings, implicitly assuming that they are the “right” ones.

    TL;DR: let’s ignore anonymous comments, let’s also ignore all rankings, and if we can’t, let’s prefer rankings that don’t push people to game them and which give high, or all, weight to peer reputation.

  47. Thanks Boaz for the link to the methodology of US news. However I am surprised that you feel that is better than [or even comparable to] that of SH. As SH say in QA “They [SH] counts the papers [of an author] toward the institution that the author is currently (according to the Brown study) a faculty member. As a result, if you hire the best people in the world, then your ranking goes up immediately. On the other hand, if you lose all your good people, then your ranking immediately goes to the bottom. We feel that this makes sense.”

    This is a very serious bug with US news ranking, i.e., it is not dynamic at all. Say a chair got his Ph.D. from a famus school X in area Y 20 years ago. This chair always think school X is the best even if there is no faulty in area Y currently there. As a result, you see very little changes in the ranking over these years and the people become disappointed with results, since even if they improve the results does not change. Of course this is in addition of lots of other issues with US news ranking, i.e., chair does not have enough knowledge about a school, the chair tries to game the system to get higher rankings for his department, etc.

  48. Thanks Boaz and Luca for sharing your thoughts. I hope it’s okay for me to post comments anonymously. I am not a well-established researcher (yet – hopefully), so I do worry that a non-anonymous comment on a controversial topic would affect my career somehow.

    In theory, we could ignore all rankings, but in practice, as some of the comments point out, people still use it for all sort of reasons. US news ranking is still the *authoritative* source but its ranking system has serious issues (as pointed out by David1 and some others) and hasn’t been improved at all.

    Realistically, I think the community as a whole has a much higher chance of success to push for a better ranking system instead of trying to convince the rest of the world ranking should be ignored.

  49. Luca, two comments:

    1) I don’t quite see how reputation rankings are complete. Care to explain that a bit further? This is not a rhetorical question. I’m genuinely puzzled by your statement.

    2) The experience from Sabermetrics (ranking of athletes and teams) is diametrically opposed to your claims. Numerical, carefully developed “fair” rankings have consistently proven to be superior metrics over subjective ones when it comes to athletes.

    Moreover, if you look at commercial rankings where individual scores are posted, usually reducing the weight of reputation seems to reduce the number of anomalies rather than increase them.

    To Boaz:

    I agree that if a single measure is used, then people try to hijack it. However the way to stop this is not to move to subjective rankings, but to use enough signals that it is not possible to hijack enough of them to make a difference. Google for example rotates between different-yet-equivalent ranking functions because any single one can be subverted by SEOs (search engine optimizers). Similarly for theory rankings, if we count number of papers, and citations, and number of papers in big conferences, and invitations to Dagstuhl, and number of PC committee memberships and number of journal publications and journal impact factor and number of ACM/IEEE/EATCS/IACR fellows and size of faculty and number of letters in last names then it is nearly as hard to subvert the ranking as it would be to assemble a good group of theoreticians to begin with.

  50. Responding to “seriously people:” what I mean is the following.

    Suppose I come up with a ranking methodology that takes into account publication, weighted by how old they are, citations, maybe weighted with some king of pagerank, and so on, with each choice having a sensible intuitive motivation, but also being one several possible choices. At the same time, Boaz, using similar data but making different choices, comes up with another ranking, which is rather different from mine.

    How do we decide which ranking is better? Because if they are “equally good,” even though they give very different results, this conclusion essentially shows that good rankings do not exist. So if a good ranking exists, then either mine or Boaz’s should be better. How do we pick? Well, maybe mine says that theory group X is better than theory group Y even though “everybody knows” that Y is better than X, for many values of X and Y, while this problem does not come up in Boaz’s ranking. Then we say that Boaz’s ranking is better.

    That is, either we agree that no ranking is possible, or else we validate different rankings by seeing how they agree with reputation, and hence we are implicitly accepting that reputation gives the right ranking.

  51. Also (again, responding to “seriously people”) the comparison to sports is interesting because this is a comparison that is often made as *criticism* of a certain way of doing research.

    But specifically to Sabermetrics I think that there is a fundamental difference: one wants a basketball player who is good at throwing the ball into the basket, or a baseball player who is good at doing whatever it is that baseball players do, and so statistics about how good one is at these things are very important. But departments don’t want to hire theoreticians who are good at publishing in STOC and FOCS, they want to hire theoreticians who are good at doing theory.

  52. Your logic is rather faulty there, Luca, when you circularly assume that the only possible arbiter of correctness is reputation.

    Let me be silly and show it by slightly modifying your example: say you come up with a multiplication algorithm and Boaz comes up with another, but they don’t agree as to the answer. So paraphrasing your words either we agree that no evaluation of correctness of multiplication is possible or else we validate different multiplication algorithms by seeing how they agree with our subjective feeling of what the multiplication ought to be.

    Of course the above is nonsense. There are methods other than reputation to score a ranking. Say, assume a certain hypothetical ranking gives a much higher score than expected to a not as well known school near Washington D.C. We can then look at their web site and examine carefully their publication record, judging work by their impact and realize, gee maybe this place is not 6th but certainly it does deserve to be much higher than what I would have given it a priori _on_a_reputation_basis_.

    So what is complete in the ranking class is not reputation, but exhaustive evaluation of each department track record.

  53. Luca, with all respect are you working in theory and is this “in theory” blog? I think using your way of arguing you could already prove P!=NP. or you could prove there should be no Google at all. Do you really claim we cannot create a reasonable ranking in this way? Then I think you can use the same argument to show there is no sport ranking as well, though there are good rankings, e.g., for soccer that changes every month. All you need is to have a committee of people to decide about important measures and their weights and can be even done in iterations (indeed I really should thank Taghi to bring the first iteration). When the measures and weights are agreed upon, then an algorithm can create a reasonable ranking as well. It is a shame that we leave this to US news to create a ranking for us with so much issues mentioned above.

  54. To “Seriously people” at 6:12pm, you are saying to count number of papers, number of citations, number of invitations to Dagstuhl, …, number of faculty.

    This is trivial to game – focus on quantity over quality in every aspect of your work (including faculty hiring). The damage to computer science is that we will have fewer and fewer major results, and more and more incremental results.

    Many other fields in science are already experiencing this problem (in part due to how NSF / NIH proposals are ranked). Those fields cannot easily escape this dynamic since they depend on NSF / NIH funding on a structural level.

    It would be a shame if computer science would inflict the same problem upon itself by creating an unnecessary ranking that is totally biased towards quantity over quality

  55. To “Seriously people” at 6:37pm, you are simply getting deeper into issues of basic science and philosophy of science without making progress.

    Mathematics is a scientific field. We know that multiplication and other basic axioms and definitions of math are “right” because we can validate them experimentally against physics (in particular against classical mechanics).

    Ranking CS departments is not a scientific field. Different people will have different opinions. I think Luca’s comments above pretty much get to the essence of things.

  56. 1) I didn’t endorse US News ranking. I think Luca eloquently explained why rankings are in general fairly useless for making important personal decisions, and so I judge a ranking mostly by how much damage it can cause when people try to optimize for it. In that I see US News as less problematic than paper counts.

    2) Re sabermetrics, there is a huge difference between sports and science which is that the goal of a sports team is to win, and so there exists a computable objective function. There is no such function for scientific progress. In that sense, I prefer the US News too because it at least measures something tangible – reputation among a certain set of people. You can claim that this set of people is clueless or malicious, and you should have asked a different set, but at least the US news ranking has more relation to reputation than the SH ranking has relation to research productivity. Similarly, search engines also have very objective metrics they need to maximize – clicks, revenue, etc..

    3) If you insist on introducing a new ranking, you should start by defining what is its objective – is it to help graduate students decide where to go to school? is it to predict where graduate students will actually go? is it to help deans measure progress that departments made? what kind of progress? do you want to quantify contributions to science? if you have some kind of goal for the ranking then you can try to understand which (if any) methodology is suitable for that goal. Also, if you create a new ranking X and claim it is “better” than ranking Y then you should give some evidence for that claim.

    4) Just as a final note, when I started as a faculty at Princeton, I remember people telling me that the US News rankings are junk but we are all waiting for the NRC ranking which will be the “gold standard”. Then the NRC came and as far as I remember, most people thought it was even worse than US News. So, I think all rankings are problematic, exactly because there is no objective truth.

  57. Luca, as to your sports analogy it is also misguided. Teams do not want to hire athletes that are good throwing the ball in the basket or running the bases. They want to hire athletes that win games, which is not something you can measure on an individual basis.

    There are many ways in which an athlete can help a team win: hitting home runs (Albert Pujols), getting himself on base with a weak hit but on the basis of running speed (Ichiro Suzuki), having a small strike zone and exceptional base stealing skills (Rickey Henderson), amazing defensive skills (Ozzie Smith), etc.

    Same as with a theory group. You can be a great theoretician because you wall paper the STOC/FOCS halls every year with your work, or maybe you publish sparsely but every time you issue a paper the field stops and takes notice (I know at least two Turing award winners who fit this description), or you may start research in many areas pointing theory to where the next great thing is. Or maybe you cracked a long standing open problem so well known that you were invited to present it in STOC even though you had missed the deadline for submission.

    Yet sabermetricians did not throw their hands in despair and said unpossible!

    Rather they approached this systematically. How to extract the signal from the noise, how to add each of the factor to get a score for a player overall and how to add the player scores to get a score for the whole team. They are not yet done, by any means, but the tools developed are now much superior to subjective rankings by humans.

    I’m positive that automated rankings today are already superior to reputation rankings. A judicious combination of some of the factors I listed in my 6:12pm posting would give a ranking that while not agreeable to every one, would never list a department with no theoreticians as a top 10 theory school thus handily beating the US News ranking.

    In other words, it wouldn’t be perfect, but the mean error when compared against exhaustive metrics would be far smaller than the US News reputation based rank.

  58. To “Seriously people” at 7:19pm, the sport team that scores the most points wins. The department that publishes the most STOC / FOCS papers (or any maximizes any other number) is not necessarily the better department. Because of this difference, the rest of your argument does not hold.

    I have the feeling that at this point you are just arguing because you do not want to give up …

  59. To Boaz: sorry as it has been mentioned repeatedly before, like it or not, faculty and students decide based on ranking. If there was no US news ranking, I was fine. But once it exists, you should challenge it to get the right one, since schools not in the top of the ranking have great loss of faculty and students due to ranking and they cannot improve if you have only US News ranking which does not change over time. To give another example, as I have been in faculty hiring committee before, the departments may not even consider applicants from not top departments for hiring (even with the same CV of a top-school applicant). Again there are some exceptions, but this is the rule of thumb. Also note that because of US news ranking and no real challenge, there is a boosting process. The people believe in US news ranking, then they give reputation scores based on that and then then believe it again…

    Also I think you are opposed to give scores to scientific words. correct? Ok let me ask you a question because you were the chair of FOCS recently. How did you select papers? How do we select papers in our conferences in general? Don’t we write reviews but at the end give scores to them and then select the best ones according to scores and of course some discussion for borderline papers or some exceptional papers (at least I have not seen any other approach than this in my whole career)? Ok, if we can really give scores to papers, can’t we have scores for departments as well in terms of some function of papers, productivity, etc (OK not a perfect one but a reasonable one that you can improve your ranking at least)?

  60. To “toboaz” – with conferences, we ask multiple people who have some preexisting reputation in the paper’s area to assign a score.

    This is what the US News does – it calls up a lot of people with some preexisting reputation in a field and asks them to assign scores to departments.

    What you did is essentially to come up with all the scores all by yourself. (The fact that you used a computer / the internet / databases when coming up with the scores does not change this fact.) [Rest of comment deleted by L.T. — Folks, please, be nice]

  61. Boaz, I agree with the issues you raise in your four points but disagree with the conclusion.

    1) In that I see US News as less problematic than paper counts.

    I don’t see why increased efforts in networking and reputation for the deans who rank schools for US news is preferable to increased paper production. Both are equally pernicious.

    2) “but at least the US news ranking has more relation to reputation than the SH ranking has relation to research productivity.”

    There are two things wrong in that statement:

    First why you consider a good thing that it is highly correlated to a measure that is well established to be bogus. In my book that is a very good argument _against_ US News. I still don’t understand how in your mind this becomes a positive.

    Second on what basis do you declare that paper counting has low correlation to research productivity? I agree that if we were to use this measure as a goal, then in the future it would be quickly subverted and lose all correlation. But you are saying something rather different. You are stating that _today_ it has very little correlation to research productivity. This is demonstrably false. How do we know that? well we can compute the correlation coefficient between Taghi’s ranking and the US News one and it gives a correlation coefficient of 0.71.

    In fact, doing a quick manual adjusting of the USN ranking in a couple of places where I think it has clearly ranked a place too high/too low and then manually adjusting Taghi’s ranking to reduce the influence of size and we get a 0.95 correlation coefficient (!!) between the two modified rankings.

    3) Also, if you create a new ranking X and claim it is “better” than ranking Y then you should give some evidence for that claim.

    Sure, but if you claim is worse you should also provide evidence for that claim.

    4) So, I think all rankings are problematic, exactly because there is no objective truth.

    I agree on the stated fact, but disagree on the implicit conclusion. You conclude “hence we shouldn’t rank” my conclusion is “we should rank but don’t get too upset if you don’t 100% agree with every placement, rather look at the max delta between where a school is and where you think it should be, then go and look at the record, and see if you change your mind”. After this the ranking that minimizes the delta above is one we could all live with.

  62. @Anon 7:52

    If the people who were asked took the time to look at groups individually then there wouldn’t be a problem with the USN rank. The problem is that they simply repeat the reputation they computed twenty years ago back when they were students at school X just like David1 said.

    A while back a researcher who was asked by the USN for his ranking actually took the time to compute the rank and consult and verify his conclusions with colleagues, rather than just restate the reputation earned long ago. Guess what happened? a certain top three school fell out of the top ten.

  63. Yes, reputation based systems suffer from a lot of well-documented problems.

    I think by now it’s clear that replacing them with “single-decider” systems is not the answer.

  64. Written by toboaz “To give another example, as I have been in faculty hiring committee before, the departments may not even consider applicants from not top departments for hiring (even with the same CV of a top-school applicant). ”

    I cannot agree more. US-News reputation based ranking provides a positive feedback for rich to get richer. Luca, Boaz, being in a position to influence the community, do something about it. Allow the competition to be fair.

  65. OK this discussion has been long enough for me and am signing off it. I think I have made my points as clear as I can make them and frankly am a bit tired of arguing with anonymous people. If there is a valid point to be made, then I’m sure someone will be found that is willing to stand behind it.

  66. I have to call bullshit on the notion that department ranking has any effect on hiring. I have been in hiring committees at Columbia, at Berkeley and at Stanford, and I have never heard of such a thing and I could not imagine such a thing anywhere else.

    Also, let’s look at where the most recent theory hires at Berkeley received their PhD: University of Washington, University of Rome, UCLA.

  67. @Boaz: If nothing else we managed to bring back the discussion to a contest of ideas rather than name calling. Good points were raised by both sides and as many things were agreed to as were disagreed to. This is surely the sign of a productive exchange.

  68. @luca: just keep in mind that academic departments are very different.

    This topic came up in the CS blogosphere a while back, I’m going from memory here but it was observed that the top half dozen departments seemed to hire rather widely [certain La Sapienza graduate actually came up in the conversation, I kid you not🙂 ].

    On the other hand universities lower down the rank seemed to be overly biased in favor of big four graduates, including a particular department which had hired exactly no one from outside that group in many years.

  69. Luca and Boaz, please do consider the effect of selection bias. You two are very well established researchers and ranking don’t affect you at all; the institutions you are in basically would only hire “superstars” of the field (their graduate school rankings don’t matter much).

    However, that’s not the case for folks who are not superstars (by definition of superstar, that means most people); or regular schools doing hiring [as @seriously people pointed out].

    As for the issues of people making comments anonymously , please take a look at this somewhat related article: http://irevolution.net/2013/10/22/trustworthiness-and-truth/

  70. I use rankings when I get an applicant from a country I don’t know. Rankings are useless for insiders but a good initial gauge for outsiders.

  71. I agree to some extent with Luca’s and Barak’s comments.
    That said, re anonymity, I believe anonymous comments are important ingredient and a great advantage in any intellectual discussion. In fact, in an ideal world all arguments should be made anonymous, and so only the pure content/argument of the comment would be considered, instead of some political power structure.

    I thus disagree with anyone attacking or disregarding anonymous comments.

    I’m also deeply not impressed by people who leave non-anonymous comments: it has no bearing on the truthfulness of your statements whether you make them anonymous or not (except if you engage in some factual description of events, for which non-anonymity might be desired to verify the data).

  72. I’m a random PhD student, like many others over here.

    Here is what I think.

    0. I don’t care if having a ranking is good or bad, or useful or not. That is a different matter.

    1. I think MohammadTaghi’s ranking is “better” than USNews’, simply because it’s more clear; and it comes from a theorist.(I trust him and his academic honesty; don’t care what his methodology was.)

    2. Giving a ranking is scientifically meaningless and is very unprofessional and ludicrous if it comes from a scientist.
    Why? Because this is a measure. First You need to formally define the error function in your measurement . Secondly, giving a measurement without explicitly stating the amount of uncertainty is absolutely meaningless.

    3. There is no globally useful well-defined error function. I’m doing complexity theory, the error function E useful for me would be more sensitive to the number of complexity theorists available. Someone’s error E’ would depend more on algorithmic game theory faculty.

    4. Even if the whole world agrees on one error function, giving a ranking is still nontrivial. At the best case you can put departments in groups, like top 10+-3 departments, etc.

    5. MohammadTaghi is using his scientific credibility and he is trying to imply that this work is based on scientific method, or has some scientific value. Otherwise, he should publish his work anonymously.

    6. If a stupid PhD applicant is obsessed with rankings, He/She should prefer Taghi’s ranking. But, I seriously doubt the intellectuality or academic honesty of someone who is already in the community as a PhD student or higher, and takes rankings seriously. But it’s OK; History of science is rife with frauds and dishonest people who were actually smart.

    7. My set of claims look a bit inconsistent, but it’s OK. I can change my words to reduce the amount of inconsistency.

  73. “If you have a problem that you can’t solve, change the problem definition.”
    Maybe, this is what Taghi did here (too) with changing the ranking.

  74. Do you think peer review of scientific papers should be anonymous? Did you try to implement non-anonymous review, say in FOCS?

  75. Cornell is tied with u. of Minnesota for rank 57. Boaz and luca, is this still your favorite reputation ranking?

  76. OK let me add a few more comments. I hope I’ll manage to resist the temptation to reply more, and make this my last comment. (I think this probably broke my personal record of comments on a blog post.)

    1) In case I wasn’t absolutely clear, I think rankings of theory groups are generally useless since there is no global linear order, and while an approximation of a total order can be useful enough when you want to choose a restaurant for lunch, it is not useful for making major life decisions such as where to go for grad school. So, I do not “support” the US News ranking. All I said is that I see less potential danger in people trying to game the reputation based ranking than in people trying to game paper-count based ranking. By “danger” I mean damage to science and our publication process, not danger to the integrity of the ranking itself which I couldn’t care less about. I would “support” a random ranking even more strongly.

    [Again to be clear, I am not saying that there is never any situation where you might find a sorted list of departments useful. Just that the ordering you choose would depend on the situation. For example, in the paper proceedings era, the SH ranking would have been very useful if you were a carpenter who wanted to know which departments would be interested in more shelving units to stack their papers.]

    2) The anonymous commenters clearly demonstrate Luca’s point – the only reason you find a placement of Cornell at 57 ridiculous is because “everybody knows” that Cornell is a stronger school than the University of Minnesota. So, you are appealing to a reputation-based criteria. Not coincidentally, unlike the case of grad school ranking, US News used a “hard metric” based approach to produce these latest rankings – http://www.usnews.com/education/best-global-universities/articles/subject-rankings-methodology

    3) People are of course free to post anonymously and I can understand some reasons why people would want to do so. In particular beginning grad students might worry about other people judging them or, perhaps more importantly, how they would judge their own comment in 10-20 years. But one may think that more senior people (especially if they are tenured) would be willing to stand behind their words. (Of course it could well be that all the anonymous commenters in this thread are young graduate or undergraduate students.)

    4) There are many differences between conference reviews and blog comments but since people asked, we didn’t have anonymous reviews at FOCS. In fact I have never seen anonymous reviews used anywhere. I and the rest of the PC knew the names of all reviewers, and without the names the reviews would have been far less useful for us. As a byproduct of the review process, we did send out anonymized excerpts of the reviews to the authors which they may have found useful. These are also not completely anonymous in the sense that the authors knew these comments were made by someone that the PC considered to be an expert on the topic. Perhaps most importantly, we did not publish these reviews, so whatever critiques they contained on the paper were for the author’s use only. To my knowledge, in most if not all contexts where academic reviews are public (e.g. book reviews), the name of the reviewer is public as well.

Comments are closed.