Article Text
Statistics from Altmetric.com
Assessing the risk of complications or adverse events following an intervention presents challenges when they have not yet occurred. Suppose, for instance, a chronic shortage of cardiac telemetry beds has prompted a hospital to implement a new policy that places low-risk patients admitted to ‘rule out myocardial infarction’ in regular ward beds (ie, with no telemetry). After 6 months and the admission of 100 such patients, no cardiac arrests or other untoward events have occurred. This absence of harm (ie, zero adverse events) indicates a low risk, but clearly we cannot infer a risk of zero on the basis of only 100 patients. But, what can we say about the true underlying risk?
The Rule of Three (Ro3), first proposed by Hanley and Hand,1 provides an estimate for the upper bound of the underlying risk when zero events of interest have occurred through considering the lack of occurrences as moderately rare. The Ro3 estimates the risk as no greater than 3/n, where n is the number of opportunities or exposures to the risk. Technically, 3/n is obtained through assigning the probability 0.95 to measure the likelihood of realising more events than those observed. Note that the Ro3 estimate applies only when one has at least 30 observations. In the above example, the Ro3 would estimate the upper limit for the risk of adverse events associated with the new policy as 3/100 or 3%.
This estimate does not sound as reassuring as simply stating that no harm has occurred among 100 patients. On the one hand, the real risk could be zero or very close to it. On the other hand, although moderately rare, there is a 5% chance that a risk as high as 3% would have resulted in such few occurrences. For practical purposes, one would like not just the “worst case scenario” (ie, the risk could be as high as 3%) but also a sense of the most likely underlying risk, which would be much closer to zero. While the Ro3 is still applied today,2 there has been little research on the subject since its development; for a detailed review see ref. 3.
In principle, we could apply this general approach to situations where we have experienced events but simple formulas such as the Ro3 do not exist and would require computer assistance and substantial statistical knowledge to solve. Thus, comparing the risk between interventions that have different numbers of trials and adverse events remains a challenge. For example, an existing intervention may have a known frequency of 0.001 of a complication, while a recently adopted intervention for the same condition has been used 500 times and no complications have been observed. How can we draw a simple comparison between the two interventions to identify which is safer?
While the Ro3 assigns 0.95 to measure the likelihood of realising more events than those observed, Quigley and Revie4 estimate this probability through minimising the maximum expected error. While numerical methods are required to obtain their estimate, a reasonable approximation can be obtained as follows. Where we have observed x events in n trials, we use two different formulas to estimate the risk; here denoted by δ. 1
It is well known within mathematical statistics that the most accurate estimate of a probability is the Maximum Likelihood Estimator, that is, x/n.5 Simply, the most likely estimate of a probability is the proportion of occurrences observed. When x events occur in n trials, while a range of possible values exist for the true underlying probability, the most likely value is x/n. Unfortunately, this intuitively plausible estimate does not apply when x is zero. Returning to the above example, while we may well believe the risk of adverse cardiac events associated with the new policy is quite low, we cannot justify saying that the ‘most likely estimate is that there is zero risk.’ In general, saying that the most likely risk is 0/n will be inaccurate and uninformative for decision makers.
The method proposed in (1) provides a non-zero estimate when zero events have been observed, that is, 2/5n and is x/n for all other situations. As such, (1) provides a simple and consistent framework by which to compare different interventions.
Two illustrative examples are used to outline the benefit of using (1) compared with the Ro3.
Example 1: A nationwide audit was carried out on all deaths from sepsis to explore whether any were preventable through more rapid initiation of sepsis management protocols. From this, it was estimated that 1 in every 100 cases was avoidable, that is, the risk was 0.01. A single hospital has made substantial changes to management of such patients and has observed no preventable fatalities in a study of 50 patients. The Ro3 suggests that the risk is between 0 and 3/50 = 0.06. However, using (1), the best estimate of the risk is 0.008.
Example 2: A serious incident occurred in a large hospital following incorrect placement of a feeding nasogastric tube. In the past 6 months (when it is believed a similar complication also occurred), it is estimated that 500 such tubes were inserted, and as a result, the risk is estimated to be 2/500 = 0.004. A change in insertion policy was made, which required a radiology consultant to review a chest X-ray to confirm tube positioning immediately and prior to use. In the ensuing 2 months, 125 tubes were inserted with no known complications. The hospital is reviewing its procedures and is keen to assess if the added cost of the radiologist reviewing the X-ray reduces the risk. The estimate using the Ro3 is 0.024, while the estimate using (1) is 0.0032.
In both examples, we use the Ro3 and (1) for different purposes. The Ro3 suggested insufficient information had been collected to conclude that either of the new interventions were improvements. As the Ro3 assesses a ‘maximum risk’, we would require excessive amounts of data to conclude otherwise. However, (1) provided a more accurate estimate suggesting each new procedure resulted in a reduction of risk.
The Ro3 can be extended to estimate the risk for situations where events have been realised; hence facilitating comparisons between risks, such as those in example 2. However, simple accessible formulas for these situations do not exist, unlike with the method proposed in (1), which would support simple, timely comparisons.
To conclude, one key shortcoming of the Ro3 is a loss of simplicity for non-zero data and so it is difficult to use when making comparisons across different interventions. A second is, when combining probabilities, the level of pessimism is compounded beyond that specified at the 5% level of the Ro3. Thus, using the Ro3 when making comparisons between interventions, decision makers may mix methods resulting in incoherent assessments. We have proposed a simple formula, which is an accurate assessment of risk as well as coherently supporting comparisons between risks.
Footnotes
-
Contributors All three authors contributed in the following way: contributed to the paper's concept including structuring and collating data, contributed to the writing and editing of the final article, approved the final version, take ownership for the content and integrity of the paper.
-
Competing interests None.
-
Provenance and peer review Not commissioned; internally peer reviewed.