Bocconi Hired Poorly Qualified Computer Scientist

Today I received an interesting email from our compliance office that is working on the accreditation of our PhD program in Statistics and Computer Science.

One of the requisites for accreditation is to have a certain number of affiliated faculty. To count as an affiliated faculty, however, one must pass certain minimal thresholds of research productivity, the same that are necessary to be promoted to Associate Professor, as quantified according to Italy’s well intentioned but questionably run initiative to conduct research evaluations using quantifiable parameters.

(For context, every Italian professor maintains a list of publications in a site run by the ministry. Although the site is linked to various bibliographic databases, one has to input each publication manually into a local site at one’s own university, then the ministry site fetches the data from the local site. The data in the ministry site is used for these research evaluations. At one point, a secretary and I spent long hours entering my publications from the past ten years, to apply for an Italian grant.)

Be that as it may, the compliance office noted that I did not qualify to be an affiliated faculty (or, for that matter, an Associate Professor) based on my 2016-2020 publication record. That would be seven papers in SoDA and two in FOCS: surely Italian Associate Professors are held to high standards! It turns out, however, that one of the criteria counts only journal publications.

Well, how about the paper in J. ACM and the two papers in SIAM J. on Computing published between 2016 and 2020? That would (barely) be enough, but one SICOMP paper has the same title of a SoDA paper (being, in fact, the same paper) and so the ministry site had rejected it. Luckily, the Bocconi administration was able to remove the SoDA paper from the ministry site, I added again the SICOMP version, and now I finally, if barely, qualify to be an Associate Professor and a PhD program affiliated faculty.

This sounds like the beginning of a long and unproductive relationship between me and the Italian system of research evaluation.

P.S. some colleagues at other Italian universities to whom I told this story argued that the Bocconi administration did not correctly apply the government rules, and that one should count conference proceedings indexed by Scopus; other colleagues said that indeed the government decree n. 589 of August 8, 2018, in article 2, comma 1, part a, only refers to journals. This of course only reinforces my impression that the whole set of evaluation criteria is a dumpster fire that is way too far gone.

12 thoughts on “Bocconi Hired Poorly Qualified Computer Scientist

  1. Similar problems are prevalent in Korea as well. AFAIK even many universities hire new faculty members based only on journal records and ignore most conference papers, so it is common advice to publish papers (as many as you can) in journals with high impact factor, especially on the high-impact mega journals. Under the circumstance that most (T)CS-related researches are discussed in conferences, I think this unfortunate system makes people less productive. Sad story.

  2. I had a similar nightmarish experience during my PhD in Germany. The PhD program required me to have two publications as FIRST author to graduate. I had seven, so this should not really be in issue, but… I published mostly in mathematics journals, which order the authors alphabetically. Although I had collected declarations from my collaborators that asserted I was indeed the FIRST/MAIN author, the university bureaucracy decided to interpret the rule literally and wanted to keep me from getting a PhD. We had to appeal and this delayed me getting my diploma for 6 months… At least they now explicitly changed the rules.

  3. “This of course only reinforces my impression that the whole set of evaluation criteria is a dumpster fire that is way too far gone.” FWIW, I second your opinion entirely Luca. I worry about the effect that the Italian research-evaluation system has on my young colleagues and on how it affects their research priorities. Good luck to everyone!

  4. This situation is also problematic in Portugal, and I believe it is the main reason why TCS research in Portuguese universities has extremely limited visibility and impact.

  5. Hah! Reminds me how my former department reached a deal that allowed for my promotion. Not sure about the exact detail – I believe it was each STOC/FOCS paper was worth 1/2 journal paper.

  6. I imagine a world without the misuse of Scopus (or whatever for it) [1,2,3]. I imagine a world in which Luca can be a professor affiliated with the Bocconi University PhD, without having to delete an article presented at SODA. I imagine a world in which an algorithm that evaluates the quality of a researcher is always only one of the many tools in the hands of a human expert. I imagine a world in which a paper presented at one of the world’s most important theoretical computer science conferences could count more than a paper published in a minor computer journal. I imagine a world where having a higher H does not mean being a better researcher, or where we can be judged even if we have a low H. Actually, I am imagining the world in which I have grown…

  7. See my essay “Content-Oblivious Quality Measures and the Control of Academia”
    The abstract reads —

    The essay examines the common practice of relying on content-oblivious quantitative measures for the evaluation of the quality of academic research. It starts with a definition of these measures, while distinguishing the raw bibliographic data and the way it is processed (to obtain a numerical value). It exposes the hidden decisions that determine which pieces of data are used and how they are processed (i.e., which statistics is taken of it). Indeed, content-oblivious quantitative measures are obtained by an automatic processing of superficial parameters of scholarly work, and their claim for objectivity hides the arbitrariness of the decisions on which they are based, and does not allow to discuss these decisions.

    The direct consequence of relying on content-oblivious quantitative measures is reaching uninformed decisions, since professional judgment that relates to the content is replaced by a superficial measure whose relevance to the questions at stake is highly questionable. The indirect consequences of relying on these measures are even more dangerous. This practice neglects the actual content while fetishizing quantity, it replaces the academic vocation by preparation for accounting, it oppresses intellectual curiosity, encourages manipulations, and reproduces power relations within scientific disciplines and between them. The content-oblivious quantitative measures are compatible with shifting the focus of evaluation from verifying the satisfaction of threshold criteria to ranking and forming a rigid hierarchy.

    The essay presents several explanations for the popularity of these content-oblivious quantitative measures: They fit the neo-liberal order and its preference for standardized regulation procedures; they carry a seductive promise of objectivity (which is extremely tempting to modern science); they serve opportunism (in the form of intellectual laziness and escaping responsibility); they empower the academic-managerial class by providing it with mechanisms for control of the academic works (by subjecting scientific knowledge to managerial knowledge); they facilitate diffusion of business attitudes to the academic world and the domination of scientific content by (bibliometric) technology. All these phenomena are related to the rise of a new political and scientific order in which a tighter control of academia plays an important role.

  8. There is a neighbor country where to evaluate researchers they do everything you write (conferences count ~0 independendently of the conference) and moreover also like to use citation indices of journals. JACM and SICOMP are doing quite bad in those rankings, around ~50%, compared to the their ranking in our minds.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s