Article Text
Abstract
Background In an attempt to aggregate observations from clinical trials, several meta-analyses have been published examining the effectiveness of systemic, non-opioid, pharmacological interventions to reduce the incidence of chronic postsurgical pain.
Objective To inform the design and reporting of future studies, the purpose of our study was to examine the quality of these meta-analyses.
Evidence review We conducted an electronic literature search in Embase, MEDLINE, and the Cochrane Database of Systematic Reviews. Published meta-analyses, from the years 2010 to 2020, examining the effect of perioperative, systemic, non-opioid pharmacological treatments on the incidence of chronic postsurgical pain in adult patients were identified. Data extraction focused on methodological details. Meta-analysis quality was assessed using the A Measurement Tool to Assess Systematic Reviews 2 (AMSTAR 2) critical appraisal tool.
Findings Our search yielded 17 published studies conducting 58 meta-analyses for gabapentinoids (gabapentin and pregabalin), ketamine, lidocaine, non-steroidal anti-inflammatory drugs, and mexiletine. According to AMSTAR 2, 88.2% of studies (or 15/17) were low or critically low in quality. The most common critical element missing was an analysis of publication bias. Trends indicated an improvement in quality over time and association with journal impact factor.
Conclusions With few individual trials adequately powered to detect treatment effects, meta-analyses play a crucial role in informing the perioperative management of chronic postsurgical pain. In light of this inherent value and despite a number of attempts, high-quality meta-analyses are still needed.
PROSPERO registration number CRD42021230941.
- pain
- postoperative
- chronic pain
- pharmacology
Data availability statement
Data sharing not applicable as no datasets generated and/or analyzed for this study. Not applicable.
Statistics from Altmetric.com
Introduction
The International Association for the Study of Pain defines chronic postsurgical pain (CPSP) as localized pain arising altered or augmented postsurgery, for a minimum of 3 months, not attributable to other causes.1 The incidence of CPSP varies between 5% and 85% depending on the operational definition and surgical procedure, with highest estimates arising from amputations, thoracotomies, cardiac surgeries, breast surgeries, inguinal hernia repairs, cholecystectomies, hip replacements, and cesarian deliveries.2 Key risk factors for CPSP include longer surgery times, patient demographics (eg, younger, female), perioperative factors (eg, pre-existing pain, nerve injuries, and acute postoperative pain), and psychological characteristics (eg, anxiety and catastrophizing).3 4 Among the deleterious effects of CPSP are prolonged opioid use, impaired function, decreased quality of life, and increased healthcare costs.5 6
The prevention of CPSP is a top priority for research in anesthesiology and perioperative medicine.7 To this end, a number of clinical trials have examined interventions to reduce the incidence of CPSP.8–10 A major challenge in interpreting outcomes from individual clinical trials are small sample sizes. Given this limitation, meta-analyses play an important role in developing evidence-based guidelines for the perioperative management of CPSP. The inherent value of meta-analyses lies in the aggregation of outcomes across trials to achieve greater sample sizes and statistical power than the clinical trials. At the top of the evidence-based pyramid, meta-analyses play an important role in informing clinical decision making. A problem emerges, however, when meta-analyses, produced en masse, yield variable and potentially conflicting and misleading outcomes.11 To aid in evaluating the quality of meta-analyses, a number of critical appraisal tools are now available (eg, A Measurement Tool to Assess Systematic Reviews 2, AMSTAR 2).12 Applied in a variety of areas of biomedical research, these tools have provided empirical evidence for areas of improvement in future meta-analyses, yet no reviews exist within the context of CPSP.13–15
To inform the design and reporting of future studies, our objective was to examine the quality of meta-analyses focused on non-opioid interventions systemically administered during the perioperative period to reduce the incidence of CPSP. Our primary aim was to use the AMSTAR 2 critical appraisal tool to examine the overall quality of meta-analyses and determine the existence of any methodological flaws that might impact the results of these meta-analyses.
Methods
Definitions
From here onward, a ‘trial’ refers to a unique published original research article where a clinical trial was conducted and was later incorporated into a meta-analysis. A ‘meta-analysis’ refers solely to the analytical component of a ‘study,’ where a ‘study’ is a published article which included one or more meta-analysis(es) and met all other inclusion criteria (described below). The search strategy was performed at the study level, as were the quality assessments.
Search strategy
An electronic literature search in Embase and MEDLINE (Ovid) databases on November 17, 2020, for the years 2010–2020 was conducted by RHM and FMW. This search was updated on July 19, 2021, to include the Cochrane Database of Systematic Reviews. The search was limited by year to capture recent studies, published after reporting guidelines for systematic reviews were made available.16 Online supplemental table 1 shows the search strategy used in these databases. The reference lists of all eligible articles were further searched for relevant studies. There were no restrictions placed on language (although no non-English articles met the inclusion criteria, the plan for such articles was that they would be translated by researchers or physicians fluent in the relevant language). The protocol for this search and review was registered in the PROSPERO database prior to conducting this review (CRD42021230941). The protocol was not published in a peer-reviewed journal. There were no deviations from the protocol with the exception of a post hoc analysis described in more detail below. The Preferred Reporting Items for Systematic Reviews and Meta-Analyses statement was used as a guideline for the reporting of this study.16
Supplemental material
Eligibility criteria
Inclusion criteria were full-text, published studies involving meta-analyses of adult patients (>18 years of age) who underwent a surgical procedure, where the intervention was a systemically delivered non-opioid pharmacological treatment initiated during perioperative management. The outcome of interest was the incidence of CPSP; we broadened the International Association for the Study of Pain definition to include any definition≥2 months postsurgery. No restrictions were placed on the control group used in the meta-analyses (eg, comparing two active treatments or an active treatment against placebo or standard of care).
Article screening and study selection
Study selection was performed in two stages by two independent reviewers (RHM and FMW). Titles and abstracts were screened in the first stage, and full texts were screened in the second stage. Any uncertainty was resolved through discussion with a third author (JLKK).
Data extraction
Relevant papers were retrieved and entered into reference management software (Mendeley V.2.62). Two reviewers (FMW and JLKK) assessed the included studies using the AMSTAR 2 critical appraisal for systematic reviews tool. In brief, both independently reviewed the study and scored the 16 checklist items, each of which is evaluated on a ‘yes’ or ‘no’ basis, with some items having the option of ‘partial yes’.12 After scoring, discrepancies were discussed to reach a consensus conclusion regarding each quality item. The final determination and categorization of quality (critically low, low, moderate, high) was determined based on the ‘Rating overall confidence in the results of the review’ provided in Shea et al,12 with ‘high’ indicating no or one non-critical flaw, ‘moderate’ indicating more than one non-critical flaw, ‘low’ indicating one critical flaw with or without non-critical flaws, and ‘critically low’ indicating more than one critical flaw with or without non-critical flaws.12 Items 2, 4, 7, 9, 11, 13, and 15 (online supplemental table 3) were considered ‘critical flaws’ based on published guidelines; the remaining items were considered ‘non-critical flaws’.12
Analysis
Our primary analysis addressed the overall quality of meta-analyses based on AMSTAR 2 scoring.12 The frequency of missing components (ie, questions scored as a ‘no’) was examined to determine the most common elements missing from meta-analyses. For this analysis, ‘partial yes’ scores were counted as ‘yes’. The association between quality and year of publication and journal impact factor (extracted from Web of Science) were examined visually using side-by-side box plots; this was a post hoc analysis that was not included in the original protocol.
Results
Search results
After the removal of duplicates, our search yielded 2904 articles. Fifty-two full-text articles were subsequently assessed, 37 of which were ultimately excluded (online supplemental table 2). Following a review of reference lists, one additional eligible study was included. A final study was added during the preparation of our manuscript, reaching a total of 17 (see figure 1).
Quality assessment and review characteristics
Seventeen published studies conducted 58 meta-analyses addressing the effect of systemic, non-opioid pharmacological interventions for CPSP.17–33 Analyses were conducted for gabapentinoids (gabapentin and pregabalin), ketamine, lidocaine, non-steroidal anti-inflammatory drugs, and mexiletine, including a total of 81 unique, identifiable trials (tables 1 and 2). One study did not clearly identify trials that were included in the analysis of chronic pain.19
According to the AMSTAR 2 critical appraisal tool, the quality of the studies performing meta-analyses was predominantly low (n=9) or critically low (n=6) (online supplemental table 4). More than two critical weaknesses were identified in five studies (29%). Moderate/high-quality meta-analyses were limited to lidocaine (n=1)29 and gabapentinoids (n=1).18 The most common AMSTAR 2 non-critical weakness (88%) was that review authors did not ‘explain their selection of the study designs for inclusion in the review.’ Critical weaknesses included failure to assess the presence and likely impact of publication bias (53%), register a protocol (35%), provide justification for the exclusion of trials (24%), consider the risk of bias in interpretation of results (12%), and assess risk of bias from individual trials included in the review (6%). A breakdown of the AMSTAR 2 scoring is illustrated in figure 2 with further details for each study provided in table 3.
Post hoc analyses revealed trends for better quality meta-analyses in more recent publications, as well as a positive relationship between quality and journal impact factor (figure 3A,B). There was no clear relationship between quality of reporting and CPSP as a primary or secondary outcome, or if a study was focused solely on CPSP.
Discussion
Our assessment using the AMSTAR 2 critical appraisal tool indicates only one ‘high’ quality published study meta-analyzing perioperative management options for CPSP.18 This is a major concern given the importance of meta-analyses to guide the perioperative management of CPSP, which are primarily informed by underpowered clinical trials. Failure to address publication bias, lack of protocol registration, and the absence of information on excluded studies were the most common critical flaws.
According to AMSTAR 2, excluded studies should be properly accounted for by review authors ‘otherwise there is a risk that they remain invisible and the impact of their exclusion from the review is unknown’.12 The problems that arise from failing to represent excluded trials is exemplified in Clarke and colleagues,27 Clarke et al, who reported an aggregate benefit of pregabalin on the basis of only two trials but provide no justification for the exclusion of Kim et al 34—a seemingly eligible third trial that examined the incidence of CPSP at 3 months after thyroidectomy surgery.27 34 The exclusion of Kim et al 34 is a concern in light of the effect on meta-results, swinging the aggregate estimate for pregabalin reported in Clarke et al 27 from significant to non-significant (OR, 0.25; 95% CI 0.05 to 1.28; unpublished data based on our own calculation). Owing to the seminal nature of Clarke et al 27 meta-analysis, Kim et al’s34 study may simply have been overlooked and captured in later studies, which benefited from increased knowledge to generate a more comprehensive search strategy. There are also a number of potential and valid explanations for exclusion to consider that may not be readily apparent, further emphasizing the need for transparency in reporting.
Lack of protocol registration was the second most common critical flaw. The fundamental goal of protocol registration is to reduce the risk of bias through adherence to a prespecified analysis plan.12 The value of a prespecified analysis plan is inherent to all meta-analyses, but particularly important when the outcome of interest is poorly defined or lacks standardization. This issue is well known to the field of CPSP and has been raised by the Initiative on Methods, Measurement, and Pain Assessment in Clinical Trials, who previously advocated for standardization of outcomes to facilitate meta-analyses and systematic reviews.35 In practice, meta-analyses examined outcomes from clinical trials ranging from three to 12 months and sometimes aggregated across time points (+3 months),18 31 which lead to differences in effect sizes reported for the same trial36 ranging from significant31 to insignificant.18 Subtler differences in defining CPSP also led to substantial variations in effect size estimates at the individual trial level, including whether the incidence of CPSP was based on pain during movement18 27 30 or at rest.24 It is worth noting, however, limitations of protocol registration. For example, a third of systematic review registrations on PROSPERO changed or did not specify the primary outcome,37 indicating a need to evaluate actual adherence to protocols in the future.
Approximately 50% of studies did not provide an analysis of publication bias. Publication bias is important to consider because an ‘underlying tendency to selectively publish small positive studies may be compounded by the effects of lower methodological quality of small studies, a greater tendency to selectively report results, and increased clinical heterogeneity when conducted in patient subgroups’.12 The lack of attention to publication bias may reflect that a number of studies included in our review analyzed fewer than 10 trials17 19–21 30 31—a benchmark to conduct quantitative analyses.38 However, there are recent examples where more than 10 trials were included and publication bias was still not considered.23 24 33 To highlight the added value of analyzing publication bias, we reanalyzed data provided in Martinez et al.24 Martinez et al’s meta-analysis was important to the field of CPSP insofar as unreported clinical trial outcomes previously ignored were included. In doing so, benefits of pregabalin were effectively nullified. While highlighting the clinical impact of publication bias, for a variety of reasons (eg, unable to contact author), outcomes from other trials that had addressed CPSP were not accessed and therefore not included. Based on the funnel plot and Doi plot39 representation of available data (figure 4), publication bias remains a concern. This is consistent with publication bias reported by Verret et al 18 for gabapentinoids and further raises concerns with regards to the efficacy of pregabalin to reduce the incidence of CPSP.
Although not the focus of our review, it was observed that few studies performed individual patient data (IPD) meta-analyses. IPD represents the gold standard for conducting meta-analyses,40 with their primary advantage lying in the ability to account for patient-level factors,40 as well as assess data quality firsthand. Additionally, heterogeneity between trials can be reduced by analyzing outcomes consistently tracked but not necessarily made available in published material. IPD meta-analyses have been conducted to assess acupuncture,41 spinal manipulative therapy,42 placebo effects in individuals with chronic pain,43 and prognostic factors associated with knee pain.44 In the context of evaluating perioperative interventions for CPSP, an IPD meta-analysis could resolve a number of major challenges, including standardizing outcomes. Notwithstanding the challenges of IPD (eg, greater resources required, risk of selection and availability bias due to barriers in accessing IPD),45 46 planning IPD meta-analysis is an important future direction of next generation CPSP meta-research.
A descriptive review of factors associated with quality of reporting indicates an improvement over time and a positive association with journal impact factor. This aligns with the timeline for the adoption of reporting guidelines for meta-analyses, endorsement of complete and transparent reporting by higher impact journals, and increased scrutiny regarding quality.47 Critically low-quality and low-quality studies continue to be published, however, as recently as 2020, thus demonstrating that challenges remain.
Strengths of our study include the first comprehensive quality assessment of non-opioid, pharmacological, perioperative interventions for CPSP. Possible limitations of quality assessments include the subjective nature of screening, data extraction, and scoring, though we attempted to address this with two investigators implementing a validated and reliable quality assessment tool.48
Conclusion
Meta-analyses play a pivotal role in informing clinical decisions with regards to perioperative management of CPSP. Based on our systematic review, meta-analyses in the area of CPSP are missing critical methodological details, including an evaluation of publication bias, protocol registration, and the provision of excluded study details. An area for future development is IPD meta-analyses, which is needed to address patient-level factors associated with trial outcomes.
Data availability statement
Data sharing not applicable as no datasets generated and/or analyzed for this study. Not applicable.
Ethics statements
Patient consent for publication
References
Supplementary materials
Supplementary Data
This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.
Footnotes
RHM and FMW are joint first authors.
Twitter @StephanKSchwarz
Contributors RHM: Responsible for study design and protocol, screening, data extraction, data interpretation, drafting the manuscript, and final manuscript approval. FMW: Responsible for study design and protocol, screening, data extraction, data interpretation, drafting the manuscript, and final manuscript approval. LDL: Contributed to data interpretation, revising the paper for intellectual content, and final manuscript approval. JJC: Contributed to data interpretation, revising the paper for intellectual content, and final manuscript approval. JAO: Contributed to data interpretation, revising the paper for intellectual content, and final manuscript approval. VPV: Contributed to data interpretation, revising the paper for intellectual content, and final manuscript approval. SKWS: Contributed to data interpretation, revising the paper for intellectual content, and final manuscript approval. JLKK: Contributed to study design, data interpretation, drafting the manuscript, revising the paper for intellectual content, and final manuscript approval.
Funding The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors.
Competing interests None declared.
Provenance and peer review Not commissioned; externally peer reviewed.
Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.