Next time you are asked your h-index ...

Next time you are asked your h-index, I suggest replying by pointing to related work by Ike Antkare [1].

[1] Ike Antkare. Ike Antkare one of the great stars in the scientific firmament.
April, 2010.
http://membres-lig.imag.fr/labbe/Publi/IkeAntkareSub.pdf,
retrieved November 25th, 2010.

Although some people still don't get it, the bottom line is always the same: you can't rely on a fully automatic method to evaluate intelligent entities which are aware of the evaluation method, as they are very likely to adapt to it. This is even more so when the automatic method relies on a data source which is not under the control of the evaluator(s).

Thanks to Lucas for sharing the bibliography entry.

I was quite impressed by this spoofing work so I checked Ike's H on the ISI web of knowledge (edited by thomson reuters, you certainely have access to it through your employer's network). Ike's publications do not appear there and Ike does not exist in their database. So unfortunately it seems that "free access" sources of data got spoofed, but not "for pay" ones.
Comment by maurizio Mon 29 Nov 2010 11:39:22 AM CET

Well, IMHO the point here isn't really about free (as in free beer) vs payware.

Rather, the point is what is the "authoritative" database of scientific publications that is used to compute h-index, or any other bibliometric measure. The payware services you mention happen to have databases with an entry barrier which is not based on automated document retrieval over the net as Google Scholar is. This is what is saving them from this specific "attack vector".

Obviously, establishing a privileged database as "authoritative" is hardly a solution and, when there is a for-profit company behind it, also induces serious ethical problems.

Publish or perish?

Comment by zack Tue 30 Nov 2010 04:03:14 PM CET

The Web of Science (ISI) indexes a small part of all the scientific publication, partly because of thematic choices, partly because of language restrictions (for example, the french reference in Law, which is the Dalloz journal, seem not to appear at all in their indexes), and a wealth of other reasons (for example, conferences in Computer Science are quite underrepresented). So the fact that ISI did not pick up this particular poison pill does not prove anything in the closed versus free contest: it is more a matter of human intelligence versus automated tools; the same technique can work quite well on ISI too, if one takes the time to poison it properly.

Comment by Roberto Di Cosmo Tue 30 Nov 2010 04:35:20 PM CET

Zack, I was not trying to make the point that the payware is somehow "better", but ISI has to be mentioned for the sake of completeness: a good paper needs a good bibliography and forgetting this particular data source would not be correct in this context. The fact that the source which resisted the spoofing happens to be the "for pay" one adds a layer of "wrong", which I wanted to underline.

I think we agree these numbers are mostly useless and often downright stupid. But the spoofing work you link to is not very convincing since it unfortunately fails to spoof the de facto authoritative source (unless I am mistaken about the "de facto authoritative" status of ISI in computer science).

Roberto, I agree this proves nothing in a "closed vs free" contest. But your statement that the same technique can work well on ISI too is not proven yet (unfortunately, as I wrote in my initial comment). The only proof we have is that google scholar is not reliable for h estimations (independently of the lack of significance of h).

Actually I fear this work reinforces ISI's "authoritative" status in some minds since its resilience to at least one form of attack is now proven. This is unfortunate for the reasons pointed out by Zack, and adds to the already unfortunate use of h.

Comment by maurizio Wed 01 Dec 2010 09:50:24 AM CET

Credits to Francesco Lovergine aka frankie, who linked me to these two articles on ISI web:

Title: DENSE-FLUID LYAPUNOV SPECTRA VIA CONSTRAINED MOLECULAR-DYNAMICS Author(s): HOOVER, WG; POSCH, HA; BESTIALE, S Source: JOURNAL OF CHEMICAL PHYSICS Volume: 87 Issue: 11 Pages: 6665-6670 Published: DEC 1 1987 Times Cited: 25

Title: DIFFUSION IN A PERIODIC LORENTZ GAS Author(s): MORAN, B; HOOVER, WG; BESTIALE, S Source: JOURNAL OF STATISTICAL PHYSICS Volume: 48 Issue: 3-4 Pages: 709-726 Published: AUG 1987 Times Cited: 108

http://tinyurl.com/6yetlfz

this is not as resounding as Ike Antkare, but noticeable. I hardly imagine that Stronzo Bestiale can be the name of an actual scientist - or even a person.

Comment by aghisla Wed 02 Feb 2011 11:48:15 AM CET