The second sprint summary from the PKP Minnesota Sprint, hosted by the University of Minnesota Libraries in May 2024, is now available.
Sprints involve PKP community members joining diverse groups to work on PKP software and support. The University of Minnesota Libraries hosted six working groups at the PKP Minnesota Sprint in May. This is a summary of one such group’s work.
Open Peer Review
Open peer review is an important review model currently not fully supported by Open Journal Systems (OJS). This group conducted an overview of existing open peer review implementations, conceptualized the various aspects of open peer review, and explored three possible workflows in OJS. For the next steps, we would like to have these considerations in GitHub discussions, put this functionality on the OJS development roadmap, and invite community feedback to help determine which workflow to prioritize.
Working Group Members
- Erik Hanson, Public Knowledge Project (PKP)
- Sonya Betz, University of Alberta
- Mariya Maistrovskaya, PKP
Background
Open peer review is an important review model that isn’t currently fully supported by OJS. Editorial teams are interested in publishing and sharing reviews with varying degrees of anonymity. Authors and reviewers may benefit from making reviews more visible. New models of publishing and review, such as preprint publishing, are also taking advantage of different reviewing models that could incorporate open review.
Goals
- Document different examples of open review;
- Describe how OJS currently handles open review;
- Create a GitHub discussion;
- Define next steps.
Brainstorming
Existing examples of Open Review:
- Royal Society Open Science
- PeerJ Post-Publication Review Model
- F1000Research’s peer review model
- OJS implementation by ESSM
- PLOS Open Review
- Open Research Europe
- eLife pre-print review model
Different flavours of Open Review:
- Open identities (reviewer and author identities disclosed);
- Open reports (publishing review reports);
- Open participation (post-publication comments);
- Combinations of all of the above;
- Reviewing can happen before publication, or afterwards.
The group also noted that review acknowledgments are linked to ORCID, but not to a specific submission, and thought about whether this use case is out of scope. Further, post-publication commenting was on the table but felt like a separate discussion, as this may be out of scope for open review.
Different aspects of open review
| Identity | Who | Model |
| anonymous | A. author | anonymous author / anonymous reviewer (1A, 1B, 2C) |
| private | B. reviewer | visible author / anonymous reviewer (2A, 1B, 2C) |
| public | C. editor | anonymous author / visible reviewer (1A, 2B, 2C) |
| D. public | visible author / visible reviewer (2A, 2B, 2C) |
The group also considered review materials such as review reports, author responses to review, editorial decisions, public comments, and revisions of articles in terms of models and timelines (e.g., pre-publication and post-publication.
Describe the potential workflows for an open reviewing process
What can OJS already do with Open Review?
- Visibility
- Existing peer review allows for three types of review: 1) anonymous reviewer/anonymous author, 2) anonymous reviewer/disclosed author, 3) open;
- Open in the sense that names and comments are visible but not made publicly available nor explicitly licensed;
- Open review allows:
- the author’s name to be visible to the reviewer within the OJS editorial backend when viewing submission metadata;
- The reviewer’s name, comments to the editor (in OJS), and uploaded review files to be visible to the author when viewing in the OJS editorial backend.
- Prior art
- Experimental plugin from the previous sprint;
- Displays list of reviewers and links out to comments from article landing page.
What new functionality would be needed:
- Sharing review materials post-publication (review process already complete)
Workflow editorial decisions for each submission: What review materials will be shared publicly from the review process? Will the publicly shared materials be anonymous or signed? - Post-publication invited / formal review (review has not occurred pre-publication)
Requires significant restructuring of workflow processes and decision-making points around revisions, the status of the article, and labeling. Phase 2! - Post-publication commenting (community reviews/discussion)
Can be tackled independently of the open review work.
Results
- We have defined the “world of possibilities” for open peer review and what questions we should consider when thinking about implementing open peer review in OJS;
- We have conceptualized the various aspects of open peer review and a variety of potential “models” that have been implemented elsewhere;
- We explored three different possible OJS workflows for open peer review, including:
- Sharing review materials post-publication (i.e. after review has been completed pre-publication);
- Post-publication invited/formal review process where review has not occurred pre-publication;
- Post-publication commenting (community reviews and discussion);
- Assessed the next steps PKP and its community could follow when implementing open peer review in OJS.
Next Steps
The goal of the sprint is to complete short projects before the end, but often there are leftovers to explore and the results become the starting point for future work.
- Capture this discussion and considerations in GitHub discussions;
- Put this work on the roadmap and consider how to make this happen;
- Invite feedback from journal editors and reviewers to determine which approach to implement and which functionality to prioritize;
- While preferences may vary by discipline, we’d want to prioritize the approach that will be of interest to most.
Thank you
We once again thank the Sprint sponsors, host institutions, and all participants for their valuable contributions to the PKP user community. Special thanks to University of Minnesota Libraries, Library Publishing Coalition, and Crossref for their support and partnerships.
