FAQs about the Open Peer Review Module (OPRM)
FAQs about OPRM in general
Why does the Open Peer Review Module (OPRM) work through invitation only? Who can request an open review?
In its current version, there are two options to send a review invitation: the end-user, most commonly an author who self-archives her work in the repository has an additional screen in the submission workflow where she can invite potential reviewers. In addition, the repository administrators may send invitations to review any work available in the repository.
Does the OPRM allow for anonymous peer reviews?
Transparency is an essential feature of the Open Peer Review Module. All interactions between authors and reviewers are signed to facilitate an open collaboration with the aim to improve the scientific quality of the reviewed work. Evidence has showed that transparency in the review process results in more constructive comments by the reviewers.
Who may make comments on the open peer reviews available on the repository?
The OPRM allows comments by reviewers and authors of the works only. Hence, reviewers of the same work may comment and score their reviews reciprocally and equally authors of the works are entitled to comment and score the reviews received.
Future versions of the module may include the option for any end-user to comment on the peer reviews available without permissions to score them.
How many times may the author of an original work use the OPRM comment functionality in order to provide feedback to her reviewers?
In its current version, the OPRM allows the author of an original work to comment on a received open peer review just once but it is foreseen to develop the functionality so that the author may be able to have a longer open discussion with the reviewers.
How have the evaluation criteria of the open peer review form been selected?
The first question is related to the validity of a given research work and examines whether it reaches scientific standards and is ready to form part of global scientific knowledge. Otherwise, whether the reviewer considers that the work still needs revisions. This is an open question that leaves the reviewer to decide based on her criteria whether the work needs improvements in any possible aspect (methodology, clarity, presentation, use of data, use of bibliography, etc). It is similar to an editorial decision of whether a research work should be published in its present form or not. The purpose of the reviewer’s effort is to signal all weaknesses that require more attention from the authors to improve the quality and make the work a more useful scientific contribution.
The rest of the questions focuses on the work’s impact or importance. The concept of impact is distinct to the concept of validity (first question). A research work can be perfectly valid (scientifically sound), but not of great significance. This impact, importance of significance, which in the current scholarly evaluation model is assessed through citations, can be specific for the restricted scientific field of the work (second question), or even extend to other fields o the society in general (third and fourth questions).
FAQs about the OPRM Reputation Metrics
How are OPRM reputations weighted?
According to the algorithms used:
- Work reputation es una agregación ponderada de las calificaciones de sus revisiones (es decir, de las valoraciones globales), donde el peso es la reputación como revisor del investigador que la escribió. La reputación del trabajo no depende de una única revisión; se necesitan varias. Cuando no hay suficientes revisiones (al menos dos), la reputación del trabajo está sin definir.
- Author reputation is an aggregation of their work reputations (taking into account only the works evaluated at the repository). The impact of a work reputation on overall author reputation is inversely proportional to the number of its authors.
- Review reputation is a weighted aggregation of comments received (i.e. quality ratings), where the weight is the reviewer reputation of the reviewer writing the comment. The review reputation is not dependent on a single comment. When not enough comments are available, the review reputation is considered the same as of the reviewer’s author reputation.
- Reviewer reputation is the reputation of a researcher as a good review writer. It is obtained from comments on their written reviews and the similarity between reviews (if my reviews are close to someone else’s reviews, then our reputations as reviewers are close). Before enough comments are gathered, the reviewer reputation of a researcher is assumed to be the same as their author reputation, or as 50/100 (i.e. an unknown factor) if no such reputation exists yet.
- The more works per author reviewed and the larger the number of reviews and comments per work, the more “accurate” the reputation metrics are.
- For the OPRM pilot study, the author reputation is calculated on a single work.
- The impact of a work reputation on its authors’ reputations is inversely proportional to the number of authors.
It is important to note:
A full explanation of the reputation module is available at the paper: Reputation in the Academic World (http://digital.csic.es/handle/10261/130842).
In the traditional journal peer review system, a common rule is to get articles peer- reviewed by two referees, and make use of a third evaluation in case of diverging reviews. OPRM reputation module works the better the more works per author reviewed and the larger the number of reviews and comments per work. However, how many reviews/comments are considered enough for the algorithms to produce fairly accurate reputation metrics (meaning, at what point reputation variations are expected to be minimal or none)?
If we assume that the quality of authors is maintained over time, our simulations show that when the number of works gets beyond 10 the values are rather stable. However, with time the research quality of researchers changes and thus the value of reputation will reflect those changes.
If the impact of a work reputation on its authors’ reputations is inversely proportional to the number of authors, does the OPRM reputation algorithms apply recommended values (providing the coefficient ϒ in the formula of the Author reputation) as to the average number of authors per work depending on the discipline? How can multidisciplinary repositories compare works reputation metrics across research areas (for instance, OPRM reputation of a Physics paper with many co-authors and that of an Economics paper with a single author?)
The weight given to the number of authors can be changed from field to field. It is currently the same for all fields but in the future we will fine tune it per discipline.
Why is the OPRM reputation module score based on the overall quality criteria only, thus neglecting other criteria in the evaluation form such as the work’s relevance within its discipline, for other research areas and for society in general?
This is an implementation issue as the reputation model can work for any dimension. However, we intend to factor all evaluation criteria available in the evaluation form in a future OPRM version.
When are the reputation scores calculated (that is to say, does the OPRM display reputations as soon as the review/comment is publicly available on the repository or is there a lag)?
Every time an article receives an open peer review, the module makes an immediate estimation of its reputation. This calculation is approximate as it takes into account the reviews ratings only.
The process to estimate reputation s for articles, reviews, authors and reviewers takes place periodically at night through iterations of algorithms to achieve accurate global metrics.
What do the reputation score colors mean?
When the reputation score is less than 30 (out of 100) they are red. Those ranging between 30 and 60 are displayed in yellow whereas those over 60 are green. In some cases, scores with a 50 colored in grey indicate that there is not a reputation value available.
Does the free text evaluation of the review form have any impact on the overall score?
Currently it does not. The reviewer’s report is a message to the author, indicating points to be taken into consideration to improve the quality of the work. This message is open so that the community can also judge whether the opinion of the reviewer is substantiated.
Why does the OPRM reputation algorithm not take into account the identity of the reviewer (meaning, for instance, if she is a highly cited author) for the calculation of review’s reputations?
The idea is that the opinions of reputable reviewers should have more weight when aggregated with the opinions of less reputed reviewers. The reviewer reputation of each researcher is computed from the opinions of other researchers about their reviewing quality not from their citations that could potentially be used as a starting point for their author reputation. It is also important to note that reviewing is different to authoring: a good author can be a bad reviewer and vice versa.