You are here

Article Text

Article menu
Download PDFPDF
ePoster session 3 – Station 2
EP092 How effective is peer review? Measuring the association between reviewer rating scores, publication status, and article impact
  1. Anuj Patel1,
  2. Brian Sites2,
  3. Steve Cohen3,
  4. Aidan Weitzner4,
  5. Matthew Davis5,
  6. Andrew Han6 and
  7. Olivia Liu4
  1. 1Department of Anesthesiology, Dartmouth Health, Lebanon, USA
  2. 2Department of Anesthesiology, Dartmouth Health; Geisel School of Medicine at Dartmouth, Lebanon, USA
  3. 3Dept. of Anesthesiology, Neurology, Physical Medicine and Rehabilitation, Psychiatry and Neurosurgery, Northwestern University Feinberg School of Medicine, Chicago, USA
  4. 4Department of Anesthesiology, Johns Hopkins School of Medicine, Baltimore, USA
  5. 5Departments of Systems, Populations and Leadership, University of Michigan School of Nursing, Learning Health Sciences, University of Michigan Medical School, and Institute for Healthcare Policy and Innovation, University of Michigan, Ann Arbor, USA
  6. 6Department of Internal Medicine, MedStar/Georgetown, Washington Hospital Center, Washington D.C., USA

Abstract

Please confirm that an ethics committee approval has been applied for or granted: Not relevant

Background and Aims Peer-review represents a cornerstone of the scientific process, yet few studies have evaluated its effectiveness to predict scientific impact. The objective of this study is to assess the effectiveness of peer-review on measures of impact for manuscripts submitted for publication.

Methods We analyzed all submitted manuscripts with abstracts (3,327) to Regional Anesthesia & Pain Medicine (RAPM) between August 2018 and October 2021. Initially, we categorized each article by topic, type, acceptance status, author demographics, and open-access status via a double-review process. Articles were scored based on the initial peer review recommendation. With any reviewer from RAPM designating the ‘reject’ classification, we further investigated if the article was published in any indexed journal comparing total citations. The primary outcome was measured via the number of citations each article had on ClarivateTM within the last two years; the number of citations from Google Scholar was also collected, along with the Altmetric score.

Results Out of 424 articles that met our inclusion criteria for analysis, we found no significant correlation between the number of Clarivate 2-year review citations and reviewer rating score (r=0.042, p=0.47), Google Scholar citations (p=0.42) or Altmetrics (p=0.70). There was no significant difference in two-year Clarivate citations between accepted (mean 7.48, SD 8.80) and rejected manuscripts (mean 5.51, SD 5.02; p=0.39). Altmetric score was significantly higher for RAPM-published papers compared to RAPM-rejected ones (mean 24.04, 63.93 vs. 2.55, 4.96; p<0.001).

Abstract EP092 Figure 1
Abstract EP092 Figure 1

Study Flow Chart

Abstract EP092 Figure 2
Abstract EP092 Figure 2

Scatter graph plotting the number of citations as a function of reviewer quality score for articles published in Regional Anesthesia & Pain Medicine

Abstract EP092 Figure 3
Abstract EP092 Figure 3

Scatter graph plotting impact outcomes as a function of accepted journal impact factor for articles rejected from Regional Anesthesia & Pain Medicine

Conclusions The ratings from peer review did not correlate with citation counts, leaving uncertain their influence on quality and other measures.

  • Peer-Review
  • Scientific Research
  • Scientific Advancement.

https://doi.org/10.1136/rapm-2024-ESRA.165

Statistics from Altmetric.com

    Request Permissions

    If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.