Guest Post by Oded Goldreich III

[Oded Goldreich has written a new essay, on taste, subjectivity, and scientific evaluation, and he tells us a bit about it. — L.T.]

This is my third guest posting on this blog, and I think it is time I express my gratitude to Luca for allowing me to benefit from the popularity of his blog.

I believe that the current posting is far less controversial than the previous ones, although it does express opinions. But these opinions refer to very general questions that address central aspects of scientific evaluation.

The first question refers to the apparent conflict between the desire to view scientific evaluation as objective and the realization that it is inevitably subjective (since it is based on opinions — expert opinions). I argue that the same holds with respect to understanding, and that the subjective basis of both understanding and evaluation does not contradict their claim to universality (to the extent that such a claim can be made at all). Thus, I believe that the concerns regarding subjectivity are overrated.

The second main claim is that imagination plays (and must play) a significant role in the evaluation process. The point is that evaluating the importance of a work requires evaluating both its past and future influence on the field. Clearly, evaluating the work’s future influence requires imagination (i.e., imagining the future development of the field). However, also evaluating past influence requires imagination, since one may need to imagine the state of the field without this work in order to realize which developments were influenced by it (and to what extent).

The third question refers to the role of personal taste in scientific evaluation. I claim that this role is grossly overrated, and that almost all that is attributed to taste is actually not a matter of taste (provided that one uses a reasonable definition of taste).

The essay, titled “On Scientific Evaluation and its relation to Understanding, Imagination, and Taste”, is available from my web-page.

Oded

About these ads

3 thoughts on “Guest Post by Oded Goldreich III

  1. The evidence for importance should be provided in the paper, the job of evaluators should not be imagining what this paper can or may lead to but to check the claims about the importance of the work provided by the author. The burden should be on authors.

  2. Anonymous #1, in an ideal world, you are correct. However, allow me to explain why I disagree with you in terms of complexity. The authors only provide a “certificate”. It is the evaluator’s job to examine the certificate and accept it only if it holds. Of course, we require of him to not be fooled by “false” certificates of importance, at least with a great probability. In order to achieve this goal, a mathematical formulation would is certainly unknown, and probably is infeasible or even inexistent. Thus, the evaluator must turn to heuristics and approximate “models” that require some part of imagination, some part intuition and of course knowledge of the field.

    To make things worse (at least towards a unique and/or automated process of evaluation) subjectivity not only affects the outcome of each evaluation, but also affects the model that provides the best results. That non-uniformity among different people can be expressed with the simple words “there is not a single model that works for everyone”. I believe that the reason behind this is that a model has to rely on some technique or the other, whereas people’s abilities and capabilities differ. For example, a model that relies on a group of people reviewing a paper together might hinder the performance of people that prefer to work alone, or reviewing two papers, one against the other, in a process of evaluation, might put at a disadvantage people that have a deep focus but cannot multitask efficiently.

    As a disclaimer, I have never reviewed a paper. I base the above on what I have read and heard from academics that have reviewed papers.

  3. To the anonymous (of Nr 1).
    Of course the authors of a work should clarify their ideas
    (including their potential applications and importance)
    as good as they can.
    But the evaluators should still imagine things,
    both with respect to potential applications
    (esp., when the paper is new)
    but also with respect to past impact
    (when the paper is not new).
    Please see the essay for details.

    In any case, this essay is *not* concerned with the duties
    of the authors (on which I wrote in the past [1, 2])
    but rather with the nature of the evaluation process.

    Oded

    [1] On our duties as scientists — see

    http://www.wisdom.weizmann.ac.il/~oded/on-duties.html

    [2] On how to write a paper — see

    http://www.wisdom.weizmann.ac.il/~oded/writing.html

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s