PKP Minnesota Sprint Summary #2 – Open Peer Review

By PKP Minnesota Sprint Working Group "Open Peer Review" & PKP Communications
A flyer for PKP Minnesota Sprint Notes being released, number 2, Open Peer Review. The OJS, OMP, and OPS logos are in the top left corner. 

Israel Cefrin, PKP Digital Accessibility and Systems Specialist, presents at the PKP Minnesota Sprint, 2024. Israel's cat, Ozzy, is featured as the star of the show. The cat is looking thoughtfully and skeptically off to the side. 

The logos of Crossref, Library Publishing Coalition, University of Minnesota Libraries, and PKP are lined up side by side to represent collaboration.
Israel Cefrin, PKP Digital Accessibility and Systems Specialist, presents at the PKP Minnesota Sprint, 2024. Israel’s cat, Ozzy, is featured as the star of the show

The second sprint summary from the PKP Minnesota Sprint, hosted by the University of Minnesota Libraries in May 2024, is now available. 

Sprints involve PKP community members joining diverse groups to work on PKP software and support. The University of Minnesota Libraries hosted six working groups at the PKP Minnesota Sprint in May. This is a summary of one such group’s work.

Open Peer Review

Open peer review is an important review model currently not fully supported by Open Journal Systems (OJS). This group conducted an overview of existing open peer review implementations, conceptualized the various aspects of open peer review, and explored three possible workflows in OJS. For the next steps, we would like to have these considerations in GitHub discussions, put this functionality on the OJS development roadmap, and invite community feedback to help determine which workflow to prioritize.

Working Group Members 

Background 

Open peer review is an important review model that isn’t currently fully supported by OJS. Editorial teams are interested in publishing and sharing reviews with varying degrees of anonymity. Authors and reviewers may benefit from making reviews more visible. New models of publishing and review, such as preprint publishing, are also taking advantage of different reviewing models that could incorporate open review.

Goals 

  • Document different examples of open review;
  • Describe how OJS currently handles open review;
  • Create a GitHub discussion; 
  • Define next steps.

Brainstorming

Existing examples of Open Review:
Different flavours of Open Review:
  • Open identities (reviewer and author identities disclosed);
  • Open reports (publishing review reports);
  • Open participation (post-publication comments);
  • Combinations of all of the above;
  • Reviewing can happen before publication, or afterwards.

The group also noted that review acknowledgments are linked to ORCID, but not to a specific submission, and thought about whether this use case is out of scope. Further, post-publication commenting was on the table but felt like a separate discussion, as this may be out of scope for open review.

Different aspects of open review
IdentityWhoModel
anonymousA. authoranonymous author / anonymous reviewer (1A, 1B, 2C)
privateB.  reviewervisible author / anonymous reviewer (2A, 1B, 2C)
publicC. editoranonymous author / visible reviewer (1A, 2B, 2C)
D. publicvisible author / visible reviewer (2A, 2B, 2C)

The group also considered review materials such as review reports, author responses to review, editorial decisions, public comments, and revisions of articles in terms of models and timelines (e.g., pre-publication and post-publication.

Describe the potential workflows for an open reviewing process
A photo of a whiteboard showing the notes of the Sprint Working Group that describes the workflow of an Open Review Process
A photo of a whiteboard showing the notes of the Sprint Working Group that describes the workflow of an Open Review Process, also summarized in the table and paragraph above.
What can OJS already do with Open Review?
  • Visibility
    • Existing peer review allows for three types of review: 1) anonymous reviewer/anonymous author, 2) anonymous reviewer/disclosed author, 3) open;
    • Open in the sense that names and comments are visible but not made publicly available nor explicitly licensed;
    • Open review allows:
      •  the author’s name to be visible to the reviewer within the OJS editorial backend when viewing submission metadata;
      • The reviewer’s name, comments to the editor (in OJS), and uploaded review files to be visible to the author when viewing in the OJS editorial backend.
  • Prior art
What new functionality would be needed:
  1. Sharing review materials post-publication (review process already complete)
    Workflow editorial decisions for each submission: What review materials will be shared publicly from the review process? Will the publicly shared materials be anonymous or signed?
  2. Post-publication invited / formal review (review has not occurred pre-publication)
    Requires significant restructuring of workflow processes and decision-making points around revisions, the status of the article, and labeling. Phase 2!
  3. Post-publication commenting (community reviews/discussion)
    Can be tackled independently of the open review work.

Results

  • We have defined the “world of possibilities” for open peer review and what questions we should consider when thinking about implementing open peer review in OJS;
  • We have conceptualized the various aspects of open peer review and a variety of potential “models” that have been implemented elsewhere;
  • We explored three different possible OJS workflows for open peer review, including:
    • Sharing review materials post-publication (i.e. after review has been completed pre-publication);
    • Post-publication invited/formal review process where review has not occurred pre-publication; 
    • Post-publication commenting (community reviews and discussion);
  • Assessed the next steps PKP and its community could follow when implementing open peer review in OJS.

Next Steps 

The goal of the sprint is to complete short projects before the end, but often there are leftovers to explore and the results become the starting point for future work.

  • Capture this discussion and considerations in GitHub discussions;
  • Put this work on the roadmap and consider how to make this happen;
  • Invite feedback from journal editors and reviewers to determine which approach to implement and which functionality to prioritize;
  • While preferences may vary by discipline, we’d want to prioritize the approach that will be of interest to most.

Thank you

We once again thank the Sprint sponsors, host institutions, and all participants for their valuable contributions to the PKP user community. Special thanks to University of Minnesota Libraries, Library Publishing Coalition, and Crossref for their support and partnerships.