[Oded Goldreich has written a new essay, which he summarizes in the guest post below. Oded looks at the distinction between “primary” and “secondary” accomplishments, that is between doing good stuff and receiving “awards”, where “award” should be taken broadly to mean something that is given after a competition that has no other purpose than choosing a winner. Oded points to the negative effects of having important decisions, e.g. about jobs, funding and promotions, be based on “secondary” rather than “primary” accomplishments, because it pushes researchers to optimize the former, which is not good for anybody. This is a broad problem with “secondary” metrics, e.g. when schools are evaluated based on test scores or police departments based on crime statistics. (I assume we have all watched The Wire.) Oded concludes that we should abolish “awards” whenever possible; I disagree with this conclusion and I will write about it in the comments. L.T.]
The purpose of this post is to call your attention to my essay “On Struggle and Competition in Scientific Fields”, which is available from the web-page www.wisdom.weizmann.ac.il/~oded/on-struggle.html
The current essay is only remotely related to my essay “On the status of intellectual values in TOC”. So I do hope that those who misunderstood and/or disagreed with the prior essay will not hold this against the current one.
The current essay is pivoted at notions such as achievement, importance, and competition. It addresses question such as the following ones. What is the difference between struggling for achievements and competing for success? What is the effect of competitions on a scientific field? What are the specific implications on TOC?
Of course, my issue is not with the semantics of (the colloquial meaning of) the forgoing words, but rather with fundamentally different situations which can be identified by referring to these words.
Loosely speaking, by struggle I mean an inherent conflict between different people who attempt to achieve various goals and positions relative to a given setting. That is, the achievements determine the outcome of the struggle. In contrast, by competitions I mean artificial constructs that are defined on top of the basic setting, while not being inherent to it, and success typically refer to winning these competitions. That is, success is determined by the outcome of the competition.
Of course, once these competitions are introduced, the setting changes; that is, a new setting is constructed in which these competitions are an inherent part. Still, in some cases — most notably in scientific fields, one may articulate in what sense the original (or basic) setting is better that the modified setting (i.e., the setting modified by competitions). These issues as well as related ones are the topic of the current essay.
The core of the essay is Section 2.1, which provides a theoretical framework in which all these notions are discussed. This framework is used in Sections 2.2 and 2.3, which revisit familiar issues such as the evolution of the FOCS/STOC conferences, the effects of awards, and why is excessive competition bad. In particular, I trace several negative social phenomena in TOC to the growing dominance of various competitions in TOC. In Section 3, I discuss the possibility of reversing the course of this evolution and reducing the dominance of competitions in TOC.
Indeed, I have expressed similar opinions regarding the evolution of FOCS/STOC and awards in the past, but I feel that the framework presented in Section 2.1 provides a better articulation of these opinions as well as a wider perspective on them. Actually, my first, initial, and most important goal in writing this essay is to clarify to myself and to other interested readers a few issues that are quite central to our professional life. My hope that I may contribute to a change in hearts, and then to a change in reality, only comes second.
Oded Goldreich
in theory 
Just to warn that Luca’s terminology is different from mine.
It seems that what he calls “primary accomplishments” correspond to what I call “achievements”, whereas what he calls “secondary accomplishments” I call “winnings of competitions” (and refuse to see it as an accomplishment of the winner…). Furthermore, I distinguish between primary and secondary competitions, between unavoidable and superfluous ones, etc (indeed, introduce an extensive taxonomy of competitions).
Oded
Eliminating awards leads to more competition, not less, since people need to struggle harder for recognition. Eliminating awards in a field (like ToC) is a bad idea because it disadvantages the field against its competitors. This again leads to more competition within the field, because there are fewer opportunities.
Obviously, a balance needs to be struck. Right now, there are too few awards, and this disadvantages theory researchers and leads to unnecessary and negative competition.
The Godel prize was created with a specific purpose in mind: to encourage authors to get their best work turned into refereed journal publications (something that was not happening at the time, even for some of the acknowledged best work in the field). I am not sure that it had the desired impact on publication but I fail to see any other distorting effect from it.
The creation of best paper awards in TOC was a result of competition, but not within the field. It is was in response to related fields where such awards were commonplace. The claim was that in some of the necessary competitions that you mentioned (positions, etc.) the lack of awards in TOC were putting deserving people at a disadvantage versus other fields. (This argues against the “obvious” superfluity of awards that you claim.) If it just were internal to TOC I don’t see that the field would have chosen to have these awards.
I don’t see any significant distorting effect of best paper awards on individual behavior within TOC. People do not submit their papers to conferences for the purpose of winning best paper prizes so this competition is not part of their strategy. The major effect of the prizes externally is to focus more attention on the field as a whole and the individuals in it. It does focus more attention within the field on the work but focus of attention in itself is not a bad thing..
Another secondary competition that was in place for many years prior is the “special issues” of journals devoted to conferences. Again, people do not submit to conferences for the purpose of being selected for these special issues. These decisions are typically made “in the heat of the moment” but do not necessarily become evident until a couple of years later. To me, the differences between special issues and best paper awards are in degree, not in kind. Do you also have a negative view of invited special issues?
If I understand Oded’s point correctly (and he will correct me if I don’t), the problem is not just the best paper award at STOC and FOCS, but rather that STOC and FOCS are run themselves as “awards”, and that decisions about jobs, promotions and grants are influenced by such “awards” (i.e. the number, rather than the merits, of STOC/FOCS papers), so that authors start “competing” for having the most papers in, instead of “struggling” to prove the best possible results.
This results in everybody losing, because the decisions on jobs, promotions and grants are not made with the best information (to the extent that they are based, even in part, on number of papers rather than their merit — the two are correlated but not the same), the programs of conferences are not chosen to make them the best conferences for the attendees (instead they look for papers most “worthy of recognition” — the two are correlated but not the same), and researchers worry about getting their papers accepted instead of doing the best possible work (correlated but not the same).
I don’t quite agree with this analysis, but I appreciate its logic, and I think that it is this dynamic that worries Oded the most, and that his distaste for the high-profile awards like Goedel prize etc. is the conclusion (rather than the starting point) of this line of thought.
I do not really understand the distinction between choosing the best papers for FOCSSTOC (and therefore having a “competition”) and choosing the “best program”.
Since today, most major conferences are attended primarily by people whose papers are accepted, what does it mean to choose the “best program”? The best program for the people who attend is ultimately the chosen program, since the main goal of most people is to attend a conference in which they can present their work.
Also, if a paper is not “safe”, the PC can say it is not among the best papers or it does not fit in the program. What is the difference between these excuses?
Independent of whether or not it is part of Oded’s argument, I’m very unclear on the idea that jobs (in particular) are dramatically influenced by the number of FOCS/STOC papers. While a large number of FOCS/STOC papers could help in the decision to examine a candidate’s record more closely, a large number of papers isn’t what decides a job (or promotion, or grant) in my experience. Once you’re at that point you start taking a closer look, which I believe Oded would argue for and approve of. YMMV.
To the extent that the argument is that the gathering of awards (papers) is useful because it’s necessary to get noticed, I would agree that it’s one possible way of getting noticed; there are certainly other ways of getting positive notice. The point is if you can’t read every applicant’s papers you start needing some way of shrinking the pile, and as Luca points out the number of FOCS/STOC papers is at least apparently correlated with quality candidates.
I’ll be away on travel for a few days, and only glanced at the above comments. I cannot do more right now. In any case, it seems quite unfair to expect me to react to comments that are already answered in the essay, either explicitly or implicitly.
I’m also disappointed that people seem to feel that it is OK to comment on the essay without giving it a *serious* read (or no read at all…). This is indeed reminiscent to my feeling at FOCSTOC reviewers. So I guess I should not be too surprised, but still I guess I’m allowed to be disappointed.
As I wrote upfront, I can take disagreements much better than misunderstanding, and now I should add also better than not trying to understand (or even read) at all.
p.s.: Just to add that I agree with Luca’s brief summary (Comment Nr 4), but would add to the list also the phenomenon of authors writing their paper (i.e., the exposition) with the aim of getting the paper accepted, rather than addressing the community at large. (In my essay I explain why these are not the same, but as Luca would say (again) there are correlated but not the same.)
P. Beame: People do not submit their papers to conferences for the purpose of winning best paper prizes so this competition is not part of their strategy.
How do you know this? I know personally about cases where your statement is false.
Dear Paul. Your reservations re awards are addressed explicitly in my essay.
Firstly, let me stress that the considerations you list are “external” to the field (i.e., to the fields contents),
as is obvious from your own text.
Next, on the practical level of handling external forces and/or parties,
I reject the claim that this is a good strategy.
It has some minor benefits (i.e., some convenience) and bigger dangers;
so *in my opinion* the trade-off on the practical level (even wrt external) is negative.
However, whatever the trade off is wrt handling the external world,
one has to ask what’s the effect on the field proper (i.e., inside).
My essay focuses on that effect.
(Lastly, I also claim that this aspect was not given sufficient thought
by those who decided to establish these awards.)
But again, let me refer all back to my (20-page) essay.
In my non-humble opinion, the essay provided as detailed
(but indeed opinionated) analysis of all these issues.
I do believe (again non-humbly…) that also those who disagree with
some points (and especially with some conclusions) may benefit
from it.
p.s.: One issue of Paul not addressed in my essay is the status of special issue. The question is not how I perceive them (or other “awards”) but rather how does our community perceive them. I think they are “counted” too, and thus they too have some distorting effect. A distortion effect is *not* defined as an effect on the current strategies of participants, but rather as the action itself — specifically, singling out 7 papers (out of 70), when there is no clear separation (on that line) at the level of actual merits (i.e.,potential importance) is a distortion. So any competition of this sort has a significant distortion, but one may ask on how does this distortion effects the field. The answer depends on the level that the outcome of that competition affects the distribution of symbolic capital (say positions) in the field. If the field has no secondary competitions, then the effect is small…
p.s.: To MM — I explicitly distinguish (in the essay) between using various heuristics in internal procedures (e.g., trimming the list of candidates) and having such heuristics effect the outcome. If you use any heuristics of your choice but the outcome is identical to the one of a monotone primary competition, then I have no issue with you…
I regret that my hastily written p.s. is a bit confusing.
Re the 1st p.s., I meant to say that we should distinguish
the distorting effect of the competition proper
(and as I argued any competition discuss here has a significant distortion)
vs the distortion this causes in other competitions
(and/or in the distribution of symbolic capital in the field at large).
The 2nd distortion depends on how much the specific competition “counts”
(in the eyes of the judges of other competitions).
Re the 2nd p.s. — I should have ended with “I have much less of an issue with you”
(rather than “I have no issue with you”). In any case, note that this position of mine
requires that the heuristics used in trimming never trims out the “deserving” winners
(i.e., those determined by a monotone primary competition).
“I’ll be away on travel for a few days, and only glanced at the above comments. I cannot do more right now. In any case, it seems quite unfair to expect me to react to comments that are already answered in the essay, either explicitly or implicitly.”
I think it is unfair to expect a serious response from others when you aren’t willing to respond yourself. This is very disappointing if you expect to be taken seriously. I don’t think your essay addressed some of the points here very clearly.
I’ll repeat my point. Awards might not be good for the field directly, but they are good for people in the field. The field as a whole needs ways of supporting good researchers in this way. It will happen one way or another, and right now in TCS a lack of “official” recognitions means that the good ol’ boy network ends up dominating in these decisions. Instead of being in the open, the competition is resolved by cronyism. This is the situation in the US.
It would be convenient if the readers who wish to comment anonymously each picked a pseudonym for the sake of this thread, so that it becomes clear if someone is following up on an earlier comment, or commenting for the first time. (If you can’t think of a pseudonym, I recommend names of Republican presidential candidates.)
Dear Oded, I am very curious what you think of say, the Nobel Prize or the Field’s medal (just to take an extreme example). I am sure that there is lot of politics and gaming involved there, which many people in those areas find quite disgusting. And this surely has side-effects like promoting trendy research, neglecting other areas and so on.
But in the bigger picture, is that really so bad? For example, these big prizes play the role of publicizing and glamorizing science to the common man, attract children to science, perhaps increase funding in science, … (isn’t it nice that Einstein is glamorized like
a pop-star, even though this is probably unfair to several other great physicists and mathematicians).
I feel that the conference awards serve a similar purpose, though on a much smaller scale of course (to the kind of benefits that Paul B. talks about). Is this really so bad?
Oded: I am sorry. I had read the footnote in your paper discussing awards but I was focused on the circumstances of the origination of the awards and neglected to answer your note there. (I should own up to the fact that I was on the committee that determined the parameters under which the best paper awards for FOCS were created. There was already an overwhelming vote of a FOCS business meeting to create such an award or awards.) The health of the field depends on its members being able to find employment that permits them to work in it, so the external cannot entirely be separated from the internal. There is something of a zero-sum game between TOC and other CS fields in hiring decisions. I have no personal evidence to claim an actual benefit for TOC hiring based on these awards but given my observations on selection of candidates for interviews/hiring between other areas, such an impact seems highly plausible. .
Any form of selection drives attention to a subset of work in a field. It therefore “distorts” the field. I do not find that inherently bad. As I see it, from your comments on special issues, you object to the small number being singled out.
My argument on awards was because they had aspects of competition that weren’t conveniently described by your dichotomies: They are clearly direct, primary, and explicit but are of a different kind from the conference submission competitions which also have these properties. I think that there should be a further distinction in competitions to distinguish ones that involve explicit versus implicit entry. (Your explicit/implicit distinction is in the competitions not the entry decisions.) My point was that one should not be concerned about the skewing effect of competitions that do not require explicit entry since they merely involve attention.
By the way, I do find your concern about competition for conference acceptance to be a more problematic issue for the field. Part of the reason that I did not refer to it is that I have not fully formulated my own opinion on the subject. I understood even before Luca’s comment that this was your primary and initial concern and didn’t mean to undercut that concern. According to economists, all human behavior has a game theoretic aspects and the question is how to design mechanisms that encourage individuals and society to behave as we would like. Can one create a mechanism that would encourage a focus on communication and the “struggle” as you term it? (Given some of the concerns you express – which I share to some extent – one could imagine a PC scoring system that explicitly took into account a rating of how well a paper communicates its results to a general TOC audience rather than to a narrow technical area.)
I managed to get some sporadict access during my travel,
took a fast look at some cmments, and am composing a brief answer off-line.
1. I’m really on travel, and I fail why this is perceived as unfair.
In any case, I did not invite a blog-discussion, let alone promised
to participate it it; I invited people to *read* my essay, and think
about it. A discussion can come later, but I don’t believe in blog
debates; my impression is that thwy tend to be merely texual fights.
I’m interested in understading.
2. I believe that my essay provides a quite extensive analysis
of some central issues, but it does not address all issues.
Its perspective and focus is the internal dynamics of fields,
and the claim is that they can and should be analyzed as such
rather than mixed with personal and external considerations.
The latter are not ignored, however. E.g., it is explicitly
stated (although one may disagree of cource) that awards do not
serve the field and/or its participants in the dealings with
outsiders (i.e., see next issue).
3. The benefit gain in external dealings when having an “awarded” case
are smaller than the loss causes when we argue for a case that has no
awards. In general, bear in mind that *most* excellent reserachers
and excellent reserach do *not* get awards! So, talking practically
and out of some experience, I fail to understand how people may
argue that awards benefit *most* of the community. They definitely
benfit only a tiny mnority of the community, a minority that is
very small also in comparison to all excellent people.
4. In cont to (3), with refernce to out-CS dealings, note that
the main complain that outsiders have is that CS (or TCS) candidates
have (relatively) few *journal* publication. The CS/TCS advocates
answer that such is the culture of our field, and they could have
answered the same wrt awards! But note that when we say that we
do not value journal publications as other fields, we are definitely
admitting some fault (even if we can justify it or xcuse ourselves).
We would need offer no apologies when saying that we have no awards;
on the contrary, we can even feel and project being “right”.
5. Awards do not promote good research. Good research is promoted
by the field’s inherent logic (please see my essay for details).
There is no need to “highlight” great achievements, they are
highlighted by their own merits, and highlighting them above
their own merits is wrong (and comes at the expense of other
good research that is being down-graded!). See my essay for more!
6. TOC Researcher asked about the public effect of Fields Medal,
Nobel Awards, and Einstein being a rock star. (Note that this is
an external consideration; clearly, from the internal perspective,
the previous item (5) deems all of these as damaging!)
I admit that there is an appealing benefit in popularizing science
via such things, but the question is of the cost (as well as actual
bewnefit), both cnsidered here only from the external perspective.
I claim that here too (as in Item (3)), at the last account, the
trade off is negative. Although at times we catch the eyes of a smart
child (which we could have done also otherwise — if we try harder…),
the “benefits” does not lead to a real understanding of what science
is about (I’m not talking of its details!) but rather to a superficial
view that prmotes no good. In analogy, like in Lord of the Ring
(or the Niberlungen Ring), although it is tempting to try to use
a powerful tool for the good — some things cannot be used that way.
[The last sentence is a metaphor; please don’t claim that this
is not a sound “proof”… :-)]
OK. I have to finish.
I wish to ask for a favour.
I really don’t enjoy this blong discussion
and I do not think it promotes any good.
Thus, I ask to be execused of it,
and will try not to look at it again
(so not to spoil the rest of my travel).
I would, however, be delighted to communicate
with anybody who wishes to communcate with me
(and will not object to that person posting our
correspondance in public).
P.s.: I forgot to answer one important comment.
It was by an annonymous claiming that the
lack of “official” recognition means that
the “old boys network” dominates.
Well, who do you think dominates the awards???
But seriously, according to my analysis (see essay),
in an autonmous field credit and recognition
is in perfect correlation with achivements.
You may disagree with this analysis,
but you cannot ignore it and say that the essay
does not address this issue.
You may also say that currently TOC is not
autonmous but rather dominated by a class that
bases its power on external forces
(there is no other alternative if you accept my analysis),
but then you have to say why you think that having more
awards will disolve this problem. In my opinion,
common sense indicates quite the opposite — it is far easier
for a small class to seize control of some central mechanisms
(e.g., awards) than to influenve the field in other way.
Awards is a rather old mechanism design, embodying and reinforcing certain ideas, in particular on the role of individuals in science and society. I thank Oded for clearly indicating that it is time to reconsider the underling goals values and wisdom of that system and be aware of the fundamental drawbacks and meanings. If we were to “start from scratch”, redesigning the social framework of ToC with our current insights, I’m certain that we could find better and more just mechanisms, even having different large scale views and values. Let’s start.
Some people need 20 pages of analysis and a public discussion before they can start to understand something that other people know instinctively after just 2 minutes of talking with a regular FOCS/STOC participant.
Interestingly, the first kind of people are or have been regular FOCS/STOC participants, whereas the second kind of people frequent other meetings.
But then, maybe the second kind of people are just losers.
—Just a thought, as I am about to decide not to read Oded’s report, which I have just printed.
I just took a look and am happy to see that the blog discussion
did not go much further. Still, I wish to address the joking
comment placed on my birthday (Feb 4th). I would have preferred
to answer the commentor in person, but since I cannot do so —
I feel forced to place my answer in public.
I find it a bit too hasty to cast judgment on a conference
and/or a community after two minutes, but maybe a couple of
hours would have allowed a wise observer to see the illness.
Still it is one thing to see an illness and another to analyze
its sources, and yet another to articulate this analysis.
I felt I need 20 pages (or so) to express what I want to say
so that it is accessible to an as wide a set of readers as possible,
and still minimize the danger of being misunderstood.
I assume one can write a 2-page digest that makes justice to this essay,
but I may not be the best person to do it (for a variety of reasons).
It has attempted to show that complexity classes P and UP are different, making proving the existence of one-way functions and responding to the problem of P versus NP millennium. See it in post “P versus UP” at the address:
http://the-point-of-view-of-frank.blogspot.com/
A paper has been published about P versus NP in IEEE-AL. Follow the arguments in my blog with some comments for dummies and the corresponding links of the published paper and the preprint(in spanish and english):
http://the-point-of-view-of-frank.blogspot.com/