6.1 Assessing the heterogeneity of your pooled effect size

Thankfully, once you’ve already pooled your effects in meta-analysis using the metagen(), metabin(), or metacont function, it is very easy and straightforward to retrieve the three most common heterogeneity measures that we described before.

In Chapter 4.2.2, we already showed you how to conduct a random-effect-model meta-analysis. In this example, we stored our results in the object m.hksj, which we will use again here.

One way to get heterogeneity measures of my meta-analysis is to print the meta-analysis (in my case, m.hksj) output again.

print(m.hksj)
##                           SMD            95%-CI %W(random)
## Call et al.            0.7091 [ 0.1979; 1.2203]        5.2
## Cavanagh et al.        0.3549 [-0.0300; 0.7397]        6.1
## DanitzOrsillo          1.7912 [ 1.1139; 2.4685]        4.2
## de Vibe et al.         0.1825 [-0.0484; 0.4133]        7.1
## Frazier et al.         0.4219 [ 0.1380; 0.7057]        6.8
## Frogeli et al.         0.6300 [ 0.2458; 1.0142]        6.1
## Gallego et al.         0.7249 [ 0.2846; 1.1652]        5.7
## Hazlett-Stevens & Oren 0.5287 [ 0.1162; 0.9412]        5.9
## Hintz et al.           0.2840 [-0.0453; 0.6133]        6.5
## Kang et al.            1.2751 [ 0.6142; 1.9360]        4.3
## Kuhlmann et al.        0.1036 [-0.2781; 0.4853]        6.1
## Lever Taylor et al.    0.3884 [-0.0639; 0.8407]        5.6
## Phang et al.           0.5407 [ 0.0619; 1.0196]        5.4
## Rasanen et al.         0.4262 [-0.0794; 0.9317]        5.3
## Ratanasiripong         0.5154 [-0.1731; 1.2039]        4.1
## Shapiro et al.         1.4797 [ 0.8618; 2.0977]        4.5
## SongLindquist          0.6126 [ 0.1683; 1.0569]        5.7
## Warnecke et al.        0.6000 [ 0.1120; 1.0880]        5.4
##
## Number of studies combined: k = 18
##
##                         SMD            95%-CI    t  p-value
## Random effects model 0.5935 [ 0.3891; 0.7979] 6.13 < 0.0001
## Prediction interval         [-0.2084; 1.3954]
##
## Quantifying heterogeneity:
## tau^2 = 0.1337; H = 1.64 [1.27; 2.11]; I^2 = 62.6% [37.9%; 77.5%]
##
## Test of heterogeneity:
##      Q d.f. p-value
##  45.50   17  0.0002
##
## Details on meta-analytical method:
## - Inverse variance method
## - Sidik-Jonkman estimator for tau^2
## - Hartung-Knapp adjustment for random effects model

We see that this output already provides us with all three heterogeneity measures (and even one more, H, which we will not cover here).

• $$\tau^{2}$$, as we can see from the tau^2 output, is 0.1337.

• $$I^{2}$$ is printed next to I^2, and has the value 62.6%, and a 95% confidence interval rangin from 37.9% to 77.5%.

• The value of $$Q$$ is displayed next to Q under Test of heterogeneity:. As we can see, the value is 45.50. In our case, this is highly significant ($$p=0.0002$$; see p-value).

• The prediction interval can be found next to Prediction interval. As we can see, the 95% interval ranges from g=-0.2084 to 1.3954.

How can we interpret the values of this example analysis? Well, all three of our indicators suggest that moderate to substantial heterogeneity is present in our data. Given the broad prediction interval, which stretches well below zero, we also cannot be overly confident that the positive effect we found for our interventions is robust in every context. It might be very well possible that the intervention does not yield positive effects in some future scenarios; even a small negative effect might be possible based on the evidence the meta-analysis gives us. Very high effect sizes, on the other hand, are possible too.

When the measures are not displayed in my output

Depending on how you changed the settings of the metagen, metabin, or metacont, it is possible that some of the measures are not displayed in your output. That’s not a big deal, because all measures are stored in the object, no matter if they are immediately displayed or not.

To directly access one of the measures, we can to use $ again (see Chapter 3.3.1). We use this in combination with our meta-analysis output object to define which measure we want to see. Code Measure$Q Cochran’s Q
$pval.Q The p-value for Cochran’s Q$I2 I-squared
$lower.I2 The lower bound of the I-squared 95%CI$upper.I2 The upper bound of the I-squared 95%CI
$tau^2 Tau-squared$lower.predict The lower bound of the 95% prediction interval
$upper.predict The upper bound of the 95% prediction interval Here are a few exmaples for my m.hksj object. As you’ll see, the output is identical to the one before. m.hksj$Q
## [1] 45.50257
m.hksj$I2 ## [1] 0.6263947 m.hksj$tau^2
## [1] 0.1337024