Distributed "Review Stack"

Distributed "Review Stack"
·
Contributors (1)
Published
Apr 18, 2019

Overview

This document imagines a distributed review and curation ecosystem where the primary activity is the awarding of badges to papers. Badges in this ecosystem are a simple dual assertion: that a paper was reviewed, and that the review conforms to a specified review policy. A badge’s meaning can be almost anything and is up to the badge provider to define. They can be used to do everything from simply tag a paper as part of a collection (ie curation), subjectively judge a paper positively or negatively (ie peer review), certify it as meeting some kind of objective criteria (ie statistical review), and beyond.

Diagram

The goal of the ecosystem should be to build at least one complete stack (on the left) for every paper — even if the ultimate badge is just that the paper doesn’t need review, or that it has been added to a collection by the badge provider.

Features

  • Posting of review policies (aka criteria) by journals and third-party reviewers

  • Assertion and depositing of reviews of a given paper (or papers) by multiple parties

  • Verification of reviews by multiple verifiers

  • Depositing of badges — a signed assertion that a review conforms to a review policy — to platforms that verify and distribute the badge.

  • Requesting badges based on history of reviews from multiple parties

  • Curating papers by posting badges with self-contained reviews

  • Cross-platform discoverability of reviews given to papers

  • Cross-platform discoverability of badges given to papers

Use Cases

Traditional Publisher

In this case, a traditional publisher provides every part of the stack, including a review policy that is referenced within the review object itself. The benefit is that providing this is simple, and requires very little work on the part of the publisher — just a few additions to article metadata. However, it might be seen as less trustworthy by the community than a stack with multiple parties involved.

Audited Review

In this case, a traditional publisher provides a badge for one of their papers, references a standard review type, and submits the badge to a review auditing service that verifies the badge conforms to their standards. Essentially, an auditor is asserting that a review was done to their standards. This provides a layer of verification that may add to community trust compared to a fully self-provided stack.

Third Party Review Service

In this case, a third party — could be a review service or a society — provides a badge for a paper published on a pre-print server based on a review they conducted and submits it to a platform. This could be part of their editorial operation, or in response to a request from an author. In this example, the review policy is the standard one published by the third party that they use for reviews of this type, but it could be self-contained or they could reference a standard type published by a review auditor.

Iterative Review

In this case, a third party — could be a review service or a society — provides a badge for a paper published on a blog based on multiple reviews published across the web by different parties and submits it to a platform. Essentially, they’re saying that the combination of all the independent reviews meets their requirements for a badge.

Editorial Curation

In this case, a third party — it could be a society or publisher — provides a badge for a paper on a preprint server that names it as part of a collection and includes a self-asserted review policy and, if they’re providing editorial commentary, a link to a review on their site. Essentially, they’re saying that they have chosen to include this paper in a collection with other papers that have been given the same badge.

Review Discoverability

Anyone can ask any platform or auditor if they have badges for a given article (initially by DOI). This allows publishers to display third-party badges on their sites, for statistical analysis of badges, etc.

Actors

Author

Someone who writes a paper.

Paper Hoster

Someone who hosts a paper. This can be:

  • Journals/publishers

  • Pre-print servers

  • Platforms

  • Individuals (ie bloggers)

Reviewer

Someone who provides a review of a paper or multiple papers. This can be:

  • Individual, independent reviewers

  • Journals/Publishers

  • Third-party review services (ie societies)

  • Bots providing automated reviews (ie statistical)

Review Policy Maintainer

Someone who maintains a review policy that, if met, merits a badge. This can be:

  • Journals/publishers

  • Third-party review services (ie society)

  • Review auditors (ie PRT)

Badge Provider

Someone who provides a badge by asserting that a review (or multiple reviews) of a paper meets the requirements of a review policy. This can be:

  • Journals/publishers

  • Third-party review services (ie societies)

  • Review auditors (ie PRT)

Badge Verifier

Someone who verifies that a badge is valid and provides a list of badges others can query. This can be:

  • Review auditors (ie PRT)

  • Platforms (ie HHMI/F1000)

  • Journals/publishers

  • Self-verified (ie for curation, where verification isn’t needed)

Objects

Paper

A scholarly work in any format, published on the web.

Paper Hosters

  • Preprint servers

  • Journals/Publishers

  • Platforms

  • Individuals (ie bloggers)

Review Policy

A human- and machine-readable description of a review process published by a policy maintainer. Essentially, a specification for a type of review. Scales from simply “a review happened” to any number of arbitrary requirements (ie non-blind with at least 1 statistical reviewer).

Policy Maintainers

  • Review auditors (ie PRT)

  • Journals/Publishers

  • Third-party review services (ie societies)

Format: JSON/XML/YAML document compiled to HTML

Locations

  • Published to review audit sites (PRT)

  • Published to journal sites

  • Published to publisher sites

  • Deposited to repository (ie Crossref)

  • Directly asserted inside reviews

  • Directly asserted inside badges

  • Linked to inside reviews

  • Linked to in JATS/paper metadata

  • Linked to from article page on publisher site in HTML metadata

Attributes

  • Name

  • Maintainer

  • List of review requirements (ie blindness, number of reviewers, etc.)

  • List of possible outcomes

Review

A human- and machine-readable assertion that a review took place published by a reviewer. Scales from simply “a single review happened” to a full version history of a multi-step review process with reviewer contacts, comments, scores, etc.

Reviewers

  • Journals

  • Publishers

  • Third-party reviewer services (ie societies)

  • Individual reviewers

  • Bots providing automated reviews (ie statistical)

Format: JSON/XML/YAML document compiled to HTML

Locations

  • Linked to in JATS/paper metadata

  • Linked to from article page on publisher site in HTML metadata

  • Deposited to repository (ie Crossref)

  • Deposited to pre-print servers

  • Published to third-party review sites

  • Published to review & curation platforms (ie HHMI platform, F1000, etc.)

  • Directly asserted inside badges

Attributes

* = required

  • Paper (or papers) reviewed (link, doi, etc.)

  • Asserter (can be anonymous/pseudonymous, can be a single reviewer, or an organization)

  • Description

  • Asserted outcome

  • Review policy (link or self-contained)

  • List of review events (ie reviewers, blindness, comments left, etc.)

    • To be further specified and somewhat standardized — but it should be designed to accommodate something as simple as “3 anonymous reviewers showed up” or as complex as showing every change to a document during a review.

    • A simplified version could be posted publicly, while a more complex version could be kept behind a paywall, functioning as subscription content.

Badge

A digitally signed assertion provided by a badge provider and verified by a bade verifier that a review or collection of reviews conforms to a review map.

Badge Providers

  • Journals/publishers

  • Third-party review services (ie society)

  • Review auditors (ie PRT)

Badge Verifiers

  • Platforms

  • Review auditors (ie PRT)

  • Journals/publishers

Format: JSON/XML/YAML document

Locations

  • Journal & publisher sites, linked from paper metadata

  • Published to third-party review sites

  • Published to review & curation platforms (ie HHMI platform, F1000, etc.)

Attributes

  • Badge name

  • Badge images (for different outcomes)

  • Badge authority (ie self, society, review auditor, etc.)

  • Link to paper

  • Review (link or self-contained)

  • Review map (link or self-contained)

  • Digital signature

Platform

A site where badges are audited, verified, and curated. Badges contain links to papers and reviews, so hosting those objects locally are not necessary, but platforms can choose to host papers and reviews.

Auditor

Any service that verifies and makes badges discoverable, including platforms.

Comments
0
comment

No comments here