Referee Feedback questionnaire

[mlwruble]

Hi All,

we've been using OJS for two years now, and and most grateful for it, thanks!

I have a request from my journal editorial board to add a questionnaire to the referee feedback page. They would like the ability for the referee to easily evaluate some review points, without having to write up a review. This would be in addition to the review comment areas that exist today.

The suggestion I've gotten, used successfully in another source is to have radio buttons, from one to ten for rating the following areas:

Significance:
Originality and novelty:
Relevance to the call for papers:
Readability and organization:
Technical quality:
Overall recommendation:
Confidence:

Before I get started learning how to modify OJS, Is there any current initiative to implement such a feature?

Assuming no to the previous question, would this be better implemented as a patch to the existing code or as a plugin somehow?

[asmecher]

Hi mlwruble,

We're getting an increasing number of inquiries on this feature, but we haven't had the chance to implement it ourselves (and nobody in the community has come forward with an implementation yet either). We'd be delighted to hear someone's working on it.

Based on the amount of interest, I think it would be best implemented as an addition to the core code rather than as a plugin. If you're able to implement it in a generalized fashion (i.e. meeting OJS code standards and providing an interface to manage the questionnaire), we'd consider merging it into our CVS repository and thus taking on maintenance.

Let me know if you have any questions. I'd be happy to provide some guidance.

Regards,
Alec Smecher
Open Journal Systems Team
---
Don't miss the First International PKP Scholarly Publishing Conference
July 11 - 13, 2007, Vancouver, BC, Canada
http://ocs.sfu.ca/pkp2007/

[jmir]

I agree - I think the ability to create a referee form is absolutely essential and the fact that OJS doesn't support this is a major limitation. If anybody has written a plugin or fix, please let me know asap.

FYI, I am posting our form for referees below. We currently ask reviewers to copy & paste this form from a word document into the two narrative forms, but this is subotpimal and leads to all sorts of problems (including TinyMCE throwing out all linebreaks - see separate threat!).

In addition there are numerous usability problems with the current design. Quite frequently, reviewers forget to make a recommendation and/or to notify the editor. The system must be redesigned so that if a reviewer enters something (all in one step) and clicks the submit button then all informaiton is on file, and the editor is notified.

To make matters worse, there is no easy way for the editor to log in as the reviewer (to complete the submission, if the reviewer has entered a narrative review but has forgotten to press the "notify" button), other than going back to the user db, search for the reviewer, log in as the reviewer etc.


JMIR Reviewer Form

-------------------------- PART I: For author and editor ------------------------------------

I. Suggestions for Authors and Editors (will be forwarded to authors) .
[When writing the Suggestions for Authors, please 1) do not mention acceptability for publication, and 2) number your specific comments]

General comments
=============
This paper....

Specific comments
=============
Major comments
---------------------
1.
2.
3.
...

Minor comments
---------------------
4.
5.
6.
...

-------------------------- PART II: For editor ------------------------------------


II.1. Confidential comments for editors

[tell us what you think and especially comment if the manuscript does not meet one of our acceptance criteria: the study conducted is ethical; the material is original; prior/related work is discussed and cited appropriately; the writing is clear; the study methods are appropriate; the data are valid; the conclusions are reasonable and supported by the data; the information is important; and the topic is interesting for our readership.]
...
...
...
...

II.2. Please note any potential conflicts of interests, which would interfere with your objectivity

II.3. If there are any parts of the manuscript, which you cannot referee because of lack of expertise in this area (e.g. statistics, English), please specify here. If you have suggestions for external reviewers, who could assess the respective part, please make them here:

II.4. Note here if you are interested in writing a editorial or commentary about this paper (outline its content)

II.5. Recommendation (please also select the respective drop-down box under 6. after submitting your narrative review under 4.):
A ( ) Accept Submission
B ( ) (Minor) revisions required
C ( ) Resubmit for (re-)review (after revisions)
D ( ) Resubmit elsewhere (or try major revision and rereview)
E ( ) Decline Submission - fatal flaws that cannot be corrected or topic unsuitable for JMIR

Hi mlwruble,

We're getting an increasing number of inquiries on this feature, but we haven't had the chance to implement it ourselves (and nobody in the community has come forward with an implementation yet either). We'd be delighted to hear someone's working on it.

Based on the amount of interest, I think it would be best implemented as an addition to the core code rather than as a plugin. If you're able to implement it in a generalized fashion (i.e. meeting OJS code standards and providing an interface to manage the questionnaire), we'd consider merging it into our CVS repository and thus taking on maintenance.

Let me know if you have any questions. I'd be happy to provide some guidance.

Regards,
Alec Smecher
Open Journal Systems Team
---
Don't miss the First International PKP Scholarly Publishing Conference
July 11 - 13, 2007, Vancouver, BC, Canada
http://ocs.sfu.ca/pkp2007/