Tuesday, September 2, 2014

What if there was a place where we didn't review results when approving publications?

I'm day dreaming about there existing a journal/book series of academic publications that encouraged improvements and innovation in methodology and data collection by letting reviewers review everything but the results when approving articles. This has actually already been done once (thanks Lilla for the tip):
These proposals will be reviewed for their importance and soundness. Once provisionally accepted, if authors complete the study as proposed, the results will be published without regard to the outcome.

Academic scholars publish their results as articles in journals, as chapters in edited volumes, as monographs, as conference oral presentations and sometimes also as in conference proceedings etc. Practically all of the time the work get reviewed before hand by anonymous reviewers, often working for free, and they approve or reject  - most often with the condition that certain revisions are made.

Many academic publications have the structure of: 
  1. introduction 
  2. background 
  3. hypothesis to be tested
  4. methodology & data 
  5. results 
  6. discussion
  7. conclusion 
(especially in the more empirical sciences). So, needless to say this kind of not-reviewing-outcome-daydreaming would only work for studies that have this kind of structure. Qualitative essays of, say, philosophy would often not really fit here. But that's ok, not everything can fit in everywhere.

You'd have 3 anonymous reviewers who review everything but the last 3 sections, and then 1 who just makes sure that the last 3 sections together with the other sections makes sense as a whole so that the editors are reassured that the judgement of the other 3 reviewers will be a good indicator of the quality of the study. Once the 3 have said that it's approved and the 1 says that it hangs together, you'd give the entire work to the reviewers and work through revisions etc. But the actual... initial approval part would be over.

Granted, this is not perfect nor is it suitable for all kinds of research. When I talk about this with people they often misunderstand and think that the results won't ever be published or in anyway reviewed, it still would. The entire study would exists at time of submission, you'd just be approved based on your hypothesis and methods. It would be reviewed as a whole and there would be revisions etc.

This would probably encourage:
a) reproduction of other peoples results 
b) publication of negative evidence in general 
c) thorough testing of predictions of theoretical models in areas where this is not so often done. 

Remember, trivial hypothesis' - no matter how sound the methodology - could still be rejected. So, any kind of studies wouldn't be approved just because the methods are good. We all know that water is wet and that Fulfulde has noun classes, just because you've got a foolproof method of proving that doesn't make it interesting to publish.

You could say that this is just a collection of grant proposals and that no reviewer would be interested in working on this.. but I think not. In a sense it's less work for the reviewers because they'll read less text and can focus on if the hypothesis and the methods makes sense. It also would be a good contribution to the advancement of science by encouraging the 3 points above.

If you could design a study and publish on it, but only be reviewed on the methods and data, not the result, what would you write about? Let us know in the comments of the blog post!

EDIT: As Simon Greenhill pointed out this is indeed very similar to the goals of journals like PLOS ONE. I really recommend reading their guidelines for reviewing and submission if you are interested in this.

No comments:

Post a Comment