Ah, right.

We thought about Journal of Irreproducible Results for a bit, and decided it would be too specialized and abstruse for that general-purpose publication.

On a serious note: Given the number of readers of a really open archive, with all their diverse backgrounds and values, I wonder if a general digg-like karma-like system for rating contributions is better. Crowds have trouble defining their terms. They’re not, in my experience, made to separate planning out a formal ontological framework with which to discuss something (like Santosh’s 7-dimensional scheme), and then sticking with it for every exemplar.

Given my experience with traditional credentialled peer-review (on both sides of the MSS), academicians aren’t very good at it, either….

As a rule of thumb, I find when designing systems like this that allowing diversity to drive emergent classification helps quite a bit.

For the promotion of higher quality literature with a more open “rating” system, there is another initiative on the internet: PLoS ONE. There’s also the Faculty of 1000.

Just look at arXiv; it has hundreds of submissions a day, and it is doing quite well as a pre-print archive for physics, mathematics, and computer science, even without a rating system.

With open preprint systems, being able to find useful and reliable ideas and data in articles is perhaps more important than being able to submit one. This becomes apparent as the number of articles increase, when searching can return hundreds and thousands of articles. One can’t go through all of them, and a few ‘bad’ articles can easily cause frustration and distrust in the quality of the submissions.

But if search criteria can include objective measures of article quality, then one can indeed easily find valuable material. Nature Precedings should therefore opt for a point-based rating system where different aspects of articles can be appraised.

Thus, instead of just letting one vote for an article, one should be allowed to rate its different aspects on, say, a 1-5 scale. Such aspects can include:

1. clarity
2. originality
3. novelty
4. presence and quality of experimental data
5. logical procession
6. depth
7. proper referencing

In effect, this would be a proper peer-review system.

The ratings, both their average and their spread, should be displayed alongside articles.

A good review/rating system will discourage submission of bad articles, build trust in the usability and reliability of content in Nature Precedings, and encourage quality submissions.

(similar comments posted elsewhere on the web by me)