SciRate is the best location I know of for public discussion and feedback on academic papers, and is an impressive open-source achievement by Adam Harrow and collaborators. Right now it has the most traction in the field of quantum informationa , but it could stand to become more popular, and to expand into other fields.
My colleague and good friend Dan Sank proposes a small but important tweak for SciRate: issue tracking, à la GitHub.
Issues in Scirate?
Scirate enables us to express comments/opinions on published works. Another very useful kind of feedback for research papers is issues. By “issue” I mean exactly the kind of thing I’m writing right now: a description of
- a problem with the work which can be definitively fixed, or
- a possible improvement to that product.
This differs from comments which are just statements of opinion which don’t require any reaction from the author. We all know that issues are essential in developing software, and based on a recent experience where I used github to host development of a research paper with three coauthors and more than a dozen group members providing feedback, I think that issues should also be used for research papers.
It might be nice to attach an issue tracker to Scirate, or at least have Scirate give links to an external issue tracker attached to each paper.
Why not just use a public github repo and get the issue tracker for free?
Making a github repo public makes everything public, including any sensitive information including comments about particular works/people. Having written a paper using github, I can imagine the authors would not want to make that repo public before going through the entire issue history making sure nobody said anything embarrassing/demeaning/etc. about anyone else or any other works. I think this is more of a problem in scientific publishing than in open source software because of the competitive nature of research and publishing in academia. As evidence, I proposed everyone in science using public github repos for their papers to a colleague, and he said he’d never do it because of the increased chance of being scooped and the obvious problems with the public seeing referee reports.
Proposal
So, it seems the best system would be for the authors to use a git repository (and an issue tracker!) for their own purposes, and to then release only curated versions of the finished work for public consumption. Of course, the repository for storing curated versions of published works is spelled “arXiv”. Therefore, I think it would be nice to have an issue tracker for scientific works which points at arXiv versions. For example:
Version 1 of https://arxiv.org/abs/1111.2890 has an error in the supplement equation 4. The denominator of the final factor should be e^{x(1-\alpha)}.
The author would respond to this issue by fixing the problem, releasing a new version on arXiv, and marking the issue as fixed.
Since Scirate already points at arXiv and is already in good use in quantum information (a field I care about), I’m wondering if it would be useful to add this sort of issue tracker to Scirate.
Problems
Who gets to close issues? Does Scirate have any way to authenticate owners of arXiv papers?
Should Scirate get into the issue tracking business? Perhaps not. If not, maybe Scirate could instead point to an issue tracker of the paper owner’s choice. This again raises problem 1 though.
As Dan has pointed out to me, this really doesn’t introduce any more moderation overhead than already exists with comments (which, just like issues, can be rebutted or simply ignored by the author). However, as he mentions at the bottom, there is (as far as I know) currently no way for authors to prove to a 3rd party that they “own” an arXiv paper, making it difficult for SciRate to determine which users can authoritatively mark issues as “fixed”, “won’t be addressed”, etc. This probably would need to be fixed on arXiv’s side through some sort of token system, which could be a very useful service to other overlay-style websites.
See also my earlier post on what the academic paper-writing process can learn from software developers.
