The Open Peer Review summary from the PKP Turin Sprint, hosted by the CRAFT-OA project in October 2024, is now available.
Sprints involve PKP community members joining diverse groups to work on PKP software and support. In October, the CRAFT-OA project and the University of Turin hosted eight working groups at the PKP Turin Sprint. This is a summary of one such group’s work.
Abstract
Open peer review (by which we mean “publication of the reviews alongside the reviewed articles”) will be integrated in OJS 3.6, but what do we do in the meantime?
An existing Ubiquity Press plugin in 3.3 and 3.4/new feature in 3.5 to allow for a MVP (minimum viable product) version of open peer review to exist in earlier versions of OJS, even if it can be helpful for getting started, the group considers it could be insufficient and additional functionality will be necessary.
Recommendations for manually managing open peer review:
- Reviews can be made accessible via the Ubiquity plugin or via supplementary files.
- Author responses should be gathered and exposed as a supplementary file.
- Pre-revision (original version) manuscript can be added as a supplementary file to complete the full constellation of work that goes into the complete “peer-reviewed publication process”
- External reviews can be “linked” as an external supplementary galley “file”
Working Group members
- Erik Hanson, Public Knowledge Project
- Clinton Graham, University of Pittsburgh
- Stephan Henn (Felix), University and City Library of Cologne
- Tim Wakeford, Ubiquity
- Jan Willem Wijnen, [Openjournals.nl](http://Openjournals.nlhttps://openjournals.nl/)
- Stefano, University of Milan
- Magnus Lu, Public Knowledge Project
Background
People in the community have shown significant interest in various aspects of open peer review, particularly regarding its integration with Open Journal Systems (OJS) and Open Monograph Press (OMP). A key focus has been linking peer review in third-party open peer review systems with the ability to publish in OJS, even when the review occurs elsewhere. Other areas of interest include exploring post-publication peer review models and creating a comprehensive inventory of what “open peer review” entails, reflecting on extensive discussions from a previous PKP Minnesota Sprint. The group follows the conversation into the various “flavors” of open peer review.
Specific technical aspects have also been highlighted, such as assigning DOIs to peer reviews and addressing copyright and licensing considerations for both the text and metadata of these reviews. The idea of copyediting published reviews and incorporating author responses has been explored, with a push for robust metadata expression to avoid relegating peer reviews to supplementary files attached to the original articles. Instead, these elements are seen as integral to an “open-first” approach that prioritizes transparency. For example, embedding peer reviews and responses in XML/JATS workflows could elevate these reviews to first-class status within the publication process. However, complications arise in mixed systems where some reviewers prefer not to have their reviews published openly.
An example of a successful open peer review platform is F1000’s ORE platform post-publication open process. Submissions undergo editorial processing and are published with reviews and author responses appearing on-site as they are completed. Reviews are assigned DOIs, and a public commenting layer fosters further engagement. Notably, this approach avoids designating a “final” published version, raising questions about compatibility with indexing systems. Another example of open peer review implementation SciPost.
Goals
- Identifying what types of open peer review can exist in a PKP/OJS context?
- Review of a preprint version, no/limited editorial involvement (pre-print open peer review)
- Standard peer review process, article accept, full peer review reports published alongside published article (post-publication review availability)
- Unsolicited comments post-publication which do not affect the editorial decisions around a publication
- This will be a lower priority for our discussion, as we want to focus on editorial decision making
- How can this be implemented in existing version of OJS/OPS? Will focus on #2 above.
- Brief aside: 3rd party peer review services
- COAR notify as e.g. link to reviews or link to overlay journal publication
- Possibility, have a way to manually indicate that a peer review has been published elsewhere, via a link
- Areas to consider:
- Visibility in frontend
- Metadata
- Ubiquity plugin that makes reviewer reports available to editors
- 💡Possibility: have plugin expose reviewers via this functionality via the frontend when labeled as “open” review in OJS (lowest hanging fruit); a new data table and backend UI element could be added to allow for publication review-by-review. Opt out must be considered, and historic data must not be accidentally exposed.
- Reviews exposed via button on article details, opt-in via “open” review in system, only reviews post-plugin installation date exposed via download button, to “opt-out,” reviewers must be changed manually by editor from “open” to some anonymous form.
- 💡Harder Possibility:To streamline manual submissions of review reports, a plugin could be written to export the metadata and review artifact in the native XML import/export format, allowing the editors to bootstrap new submissions of the reviews as published objects.
- Very mixed use cases of whether reviewers are interested in having their name associated with their review. A single form can be used with the final question of: please enter your name if you want it to be disclosed to the author and within the publication of the review.
- Author replies are not systematically captured in OJS currently. These may be uploaded as revised documents, or may be part of the discussions feature.
- 💡Possibility: Create a best practices document which describes what publishing reviews and author responses should look like in OJS. This could be used by an Editor to create new submissions which can be copyedited and published alongside the original articles.
- Does the pre-review version need to be exposed in order for the published peer review documents to be meaningful?
Results
Day 1:
- Open peer review will be expanded upon in OJS 3.6, but that’s not coming for a while and doesn’t help advance providing open peer review now. A few changes to the existing Ubiquity plugin in 3.3 and 3.4/new feature in 3.5 to allow for a MVP (minimum viable product) version of open peer review to exist in earlier versions of OJS.
- 🚧This is an OJS 3.3 and 3.4 “very hacky, probably bad, insufficient for sure” workaround to adding open peer review support. It is helpful for getting started with open peer review, but should be treated as such.
- Using existing “author friendly” review report download option, expose download link on article landing page to make “open” peer reviews visible.
- Additional functionality via plugin will be necessary to limit this functionality to after the plugin was added to limit exposure of historical peer review data
- Author responses should be gathered and exposed as a supplementary file.
- Pre-revision (original version) manuscript can be added as a supplementary file to complete the full constellation of work that goes into the complete “peer-reviewed publication process”
- External reviews can be “linked” as an external supplementary galley “file”
Day 2:
Reviewer reports
- Two options:
- Form with questions for reviewers
- Specific questions like, does it fall within scope of journal
- No form, completely open
- Form with questions for reviewers
- Common elements:
- Prose content of the review itself
- Recommendation on the review
- Reviewer identity (in one form or another) — open question: is the reviewer identity exposed? In which ways? To whom?
- Reviewer roles (e.g. Credit roles, which don’t exist quite yet)
- 🙋Is there anything interesting that ORE currently gathers about reviewers that we’re not considering?
- Metadata:
- Are there any other metadata service providers that have interesting models for reviewer metadata
- Whether a reviewer wants credit for their review contribution, e.g. with ORCID
- License info?
- Side note: Find out more information about how current (e.g. ORE) reviewer report DOI metadata is constructed
- Let authors choose between different open licenses. Does this extend to peer reviews? What about differences between peer review and article
- Type of report: e.g. refereed report, author comment, recommendation, possibly editor report as well (per Crossref)
- Stage (per Crossref) pre or post-publication
- 🙋To consider: communication that the review is structured or unstructured?
- Competing interests
- Reviewer expertise
- What happens when reviews are co-reviewed, e.g. by early career researchers and more senior researchers?
- Crossref metadata schema for peer reviews https://www.crossref.org/documentation/schema-library/markup-guide-record-types/peer-reviews/
Author comments/reports
- Content
- Article amendments. Where do they belong? They are a part of journal article versioning, but should the be presented in the reader-facing frontend as part of the “review box”
- Author identity, affiliation
- Which author name is associated with the response? Is it a singular author for all responses? Is it the “OJS user” who submitted it via OJS?
- Competing interests
- Usually author comments are to individual reviews, but it can come down to how the review reports are managed. Are they sent to the author all at once? Or individually?
- Previous review stage in OJS ends when a review ends (mostly), how do author comments fit within this? They cannot be “required,” but if they are, is a review complete when a reviewer send their review or when an author comments on it?
- Metadata
- Do they get DOIs? Author comments are supported by Crossref. Not by Datacite (as of 2024-10-09).
- Can the author response be part of the general peer review/reviewer report DOI? Maybe?
- Current goals
- Make all “open” things visible
- Improve/implement author response workflow. Make it a separate piece with its own metadata
Open peer review as described in this document is limited in scope to an invited, refereed, peer review process, and other formats (e.g. with preprints) should be explored in future work.
Next Steps
Topics to explore:
- Preprints
- What happens to OJS when a submission has already been reviewed elsewhere (OPS or otherwise)
- Prepare the day 1 guide as a blog post
- Take day 2 results to the development team
Additional technical information
https://forum.pkp.sfu.ca/t/open-peer-review-of-the-preprint-in-the-public-journal-website/74797/3
Thank you
We once again thank the Sprint host institutions, and all participants for their valuable contributions to the PKP user community. Special thanks to the CRAFT-OA Project and University of Turin for their support and partnerships.
