Roland Piquepaille wonders whether we should worry about the possibility of hundred of thousands of fake computer science papers sitting online. I don’t think we need to worry that much. I address this from the larger issue of fake scholarly papers rather than just computer science.

In this blog entry, Roland Piquepaille asks the following question:

So, is the possibility of hundred of thousands of fake computer science papers sitting online represents a danger or not? Time will tell, but please let me know what you think.

He is prompted to ask this because of the recent story about MIT students who wrote software to produce fake CS (Computer Science) papers. One of them was initially accepted by WMSCI 2005 but eventually they rejected it. Roland worries about what would happen if such fake papers are indexed by search engines (eg. Google) and artificially promoted in search results by heavy linking to them. Read his article if you want more details.

I’m going to widen the scope of discussion here beyond just CS papers and also cover academic papers in the humanities. I used to be a computer engineer but that’s not my domain anymore. My recent experience with published papers has been with materials from the humanities. I think the problem here is how to decide whether a particular paper is valid or not. I don’t think it matters much whether it is a CS paper or a paper about a topic that falls within the domain of the humanities. (If I’m wrong on this, please tell me why.)

I don’t think the possibility of producing fake papers should cause much worry. I have read some of the papers produced by the software referred to above and I must say that the papers produced don’t quite sound right. Take for instance, this paper which begins with the sentence:

Many physicists would agree that, had it not been for congestion control, the evaluation of web browsers might never have occurred.

I don’t know about you but I find this statement highly dubious. The terms congestion control and evaluation of web browsers fall well within the realm of computer science. Neither are concepts that physicists typically work with when they do physics. Now, don’t get me wrong: I’m not saying that physicists never contribute to computer science, they quite patently do, but asserting that many physicists have a professional opinion about congestion control and the evaluation of web browsers is highly dubious. Moreover, what the heck does the evaluation of web browsers mean? Ranking them against one another? Counting the number of bugs? What? Evaluation is too broad a term. Even in an abstract, such vague language is red flag that something is amiss here.

So right off the bat, the paper doesn’t look right. But one could object that it may just be that it is a legitimate paper but that it is badly written. Indeed, it could simply be a badly written paper. It could be that there is some substance to it but that this substance is marred by serious language issues. Read more of the paper above and see if you can make sense of it. It juxtaposes concepts that don’t naturally belong together but doesn’t provide a justification for these juxtapositions. On the first page: congestion control, web browsers, voice-over-IP, DHCP, XML, fiber-optic cables, low-energy, Bayesian, etc… It’s all over the map.

If I still assume that it is a legitimate paper, not a fake, based on the inane language it contains, I have to consider it to be an instance of bad scholarship. I think much more work would have to be done for such automatically-generated paper to pass as genuine scholarship in my eyes. (The WMSCI made that mistake but see this opinion of the WMSCI. The author thinks they are not serious.)

My ramblings here suggest a hierarchy:

1. genuine scholarship, clearly written
2. genuine scholarship, badly written
3. bad scholarship, clearly written
4. bad scholarship, badly written
5. fake scholarship

The order is from greater scholarly value to lesser scholarly value. I realize that there are certainly problems with the hierarchy above. (This entire blog entry is but a brain dump, so caveat emptor.) For one thing, a piece of fake scholarship might have very high value for a scholar studying fake scholarship! Also, texts are not monolithic, a specific paper may have parts that are genuine scholarship, some that are bad, some that are clearly written, some that are not. Anyway, this is a first cut.

Now, my own method is to reject anything from bad scholarship, clearly written and below. To decide where a paper falls, I have to read it and understand it. If I don’t understand it, I don’t use it. I don’t necessarily reject it either: if I feel the problem is me, not the paper, I may put it aside and seek explanations from somewhere else. The popularity of a paper may help it get on my radar map but won’t automatically make it valid in my eyes. I have to read it, period.

People, scholars or otherwise, that evaluate papers based on their popularity or visibility, or because some peer-reviewing process has vouched for them, are playing a dangerous game. I’ve seen plenty of instances of bad scholarship that for some reason, had passed the peer review process. When someone accepts a paper as valid without having read it, this person is taking the risk of basing his own work on a piece of crud. If that crud is eventually recognized for what it is, then this person’s own work may suffer from it.

So considering the problem of filtering out those fake papers, I think only people who base their evaluations of scholarly validity on popularity (of whatever kind) need be worried. People that base their opinion on reading the papers need not worry.

I know there are other aspects to the problem of fake papers but I’m not going to address them for now.