Skip to contents

Introduction

An ordination plot can suggest that groups of samples differ. ecodive’s permutation tests check whether the difference is real: does a variable, such as treatment or body site, explain the distances between samples?

Each test takes a distance matrix (dm) from any beta diversity metric, plus the variable to test: either a vector with one value per sample, or the name of a column in a data frame of sample metadata (data).

Function Test Variable Question
permanova() Permutational Multivariate Analysis of Variance either Do group centroids differ? How much variance (R²) does it explain?
permdisp() Homogeneity of Multivariate Dispersions categorical Do groups differ in their spread?
anosim() Analysis of Similarities categorical Are distances within groups smaller than between them (ranks)?
mrpp() Multi-Response Permutation Procedure categorical Are distances within groups smaller than expected?
mantel() Mantel Test numeric Do distances correlate with differences in a numeric variable, or with another distance matrix?
mantel_partial() Partial Mantel Test numeric Do they still, after removing the effect of a third distance matrix?

They give the same statistics as vegan’s adonis2(), betadisper(), anosim(), mrpp(), mantel(), and mantel.partial(), with permutations run in parallel C, and the p-values are identical for any number of threads.

The examples use R’s built-in iris data, as in vignette('ordination'): 150 flowers, four measurements each, and three species.

library(ecodive)

dm <- bray(iris[, 1:4])

PERMANOVA

permanova() is the most widely used test. Give the grouping variable as a column name of data, or as a vector.

res <- permanova(dm, 'Species', iris)
res
#> Permutational multivariate analysis of variance (PERMANOVA) of Bray-Curtis Dissimilarity
#> 
#> F = 532.7, p = 0.001, z = 577.6
#> Permutations: 999 (free)
#> 
#> Model R2 = 0.8788. Model tested as a whole:
#> 
#>           Df SumOfSqs     R2     F Pr(>F)    
#> Model      2   2.3173 0.8788 532.7  0.001 ***
#> Residual 147   0.3197 0.1212                 
#> Total    149   2.6370 1.0000                 
#> ---
#> Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Species explain most of the variation between flowers (R2), and no permutation of the labels did as well (p). The p-value cannot be smaller than 1 / (permutations + 1), so use more permutations for a more precise one.

plot() shows the pseudo-F of every permutation, and the observed one far beyond them:

plot(res)

A formula tests several variables at once. by = 'terms' tests each in turn, as adonis2() does, and strata restricts permutations to within blocks (such as subjects in a repeated-measures design).

permanova(dm, data = iris, formula = ~ Species + Sepal.Width, by = 'terms')
#> Permutational multivariate analysis of variance (PERMANOVA) of Bray-Curtis Dissimilarity
#> 
#> F =   825, p = 0.001, z =  1105
#> Permutations: 999 (free)
#> 
#> Model R2 = 0.9443. Terms added sequentially (first to last):
#> 
#>              Df SumOfSqs     R2      F Pr(>F)    
#> Species       2   2.3173 0.8788 1151.7  0.001 ***
#> Sepal.Width   1   0.1728 0.0655  171.8  0.001 ***
#> Residual    146   0.1469 0.0557                  
#> Total       149   2.6370 1.0000                  
#> ---
#> Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Checking the spread

PERMANOVA can also give a small p-value when groups have the same center but different spreads. permdisp() tests the spread on its own; a significant result means a PERMANOVA difference may come partly or wholly from dispersion.

res <- permdisp(dm, 'Species', iris)
#> Warning: Some squared distances to group centres are negative, and were set to
#> 0.
res
#> Homogeneity of multivariate dispersions (PERMDISP) of Bray-Curtis Dissimilarity
#> 
#> F = 0.5676, p = 0.573, z = -0.4401
#> Permutations: 999 (free)
#> 
#> Distances to group centroids:
#>     setosa versicolor  virginica 
#>    0.03781    0.04248    0.04103 
#> 
#>            Df  Sum Sq   Mean Sq     F N.Perm Pr(>F)
#> Groups      2 0.00057 0.0002852 0.568    999  0.573
#> Residuals 147 0.07386 0.0005024
plot(res)

Here the three species are equally spread, so the PERMANOVA result above reflects a real difference in their centers. The warning comes from Bray-Curtis distances not being Euclidean: a flower can then sit at a negative squared distance from its species’ center, which permdisp() sets to zero (Anderson 2006).

ANOSIM and MRPP

anosim() compares the ranks of distances within and between groups. R near 1 means every group is distinct, and near 0 means no difference. mrpp() compares the average distance within groups with what random groups would give.

anosim(dm, 'Species', iris)
#> Analysis of similarities (ANOSIM) of Bray-Curtis Dissimilarity
#> 
#> R = 0.8576, p = 0.001, z =  79.4
#> Permutations: 999 (free)
#> 
#> Mean rank of the distances, between and within groups:
#>            pairs mean_rank
#> Between     7500   7163.90
#> setosa      1225   2239.66
#> versicolor  1225   2457.85
#> virginica   1225   2418.15

res <- mrpp(dm, 'Species', iris)
res
#> Multi-response permutation procedure (MRPP) of Bray-Curtis Dissimilarity
#> 
#> delta = 0.05795, p = 0.001, z = -78.9
#> Permutations: 999 (free)
#> 
#> Mean within-group distances (weights: n):
#>        setosa versicolor virginica
#> delta 0.05502     0.0602   0.05861
#> n          50         50        50
#> 
#> Expected delta: 0.158
#> Chance-corrected within-group agreement A: 0.6332

plot() of an MRPP result shows each group’s mean within-group distance, with their weighted mean (the observed \delta) and the mean of all distances (the \delta expected by chance); type = 'null' shows the permutations instead, as for the other tests.

plot(res)

Pairwise comparisons

When an overall test finds that the groups differ, the next question is usually which ones. pairwise = TRUE reruns the test on each pair of groups’ samples, and adjusts the pairs’ p-values for multiple testing (Holm’s method by default; p_adjust = 'BH' controls the false discovery rate instead). The overall result is unchanged.

res <- permanova(dm, 'Species', iris, pairwise = TRUE)
res$pairwise
#>                               F        R2         z   n n_perm p_value
#> setosa-versicolor     552.51266 0.8493496 412.88785 100    999   0.001
#> setosa-virginica     1001.54509 0.9108722 688.94049 100    999   0.001
#> versicolor-virginica   91.82959 0.4837475  75.54295 100    999   0.001
#>                      p_adjusted
#> setosa-versicolor         0.003
#> setosa-virginica          0.003
#> versicolor-virginica      0.003

p_value is each pair’s own permutation p-value, and p_adjusted the value to report. anosim() and mrpp() take the same arguments, and permdisp() adds adjusted p-values to its pairwise t tests:

permdisp(dm, 'Species', iris, pairwise = TRUE)$pairwise
#> Warning: Some squared distances to group centres are negative, and were set to
#> 0.
#>                               t Df          z   n n_perm   p_value p_value_perm
#> setosa-versicolor    -1.0328277 98 -1.0325057 100    999 0.3042267        0.294
#> setosa-virginica     -0.7327500 98 -0.7079531 100    999 0.4654599        0.471
#> versicolor-virginica  0.3204074 98  0.2900848 100    999 0.7493417        0.764
#>                      p_adjusted p_adjusted_perm
#> setosa-versicolor     0.9126800           0.882
#> setosa-virginica      0.9309198           0.942
#> versicolor-virginica  0.9309198           0.942

Mantel test

mantel() correlates dm with a numeric variable, or with a second distance matrix: here, whether flowers with similar petals also have similar sepals. method can be 'pearson', 'spearman', or 'kendall'.

sepals <- bray(iris[, 1:2])
petals <- bray(iris[, 3:4])

res <- mantel(sepals, petals, method = 'spearman')
res
#> Mantel test (Spearman) of Bray-Curtis Dissimilarity and Bray-Curtis Dissimilarity
#> 
#> r =  0.69, p = 0.001, z = 35.65
#> Permutations: 999 (free)
#> 
#> Correlation: Spearman's rank correlation rho
#> Alternative: greater (r is more positive than by chance)
#> Samples: 150
plot(res)

mantel_partial() removes the effect of a third set of distances, given: here, whether sepals and petals still match after accounting for petal width alone.

mantel_partial(sepals, petals, given = 'Petal.Width', data = iris, method = 'spearman')
#> Partial Mantel test (Spearman) of Bray-Curtis Dissimilarity and Bray-Curtis Dissimilarity
#> 
#> r = 0.3918, p = 0.001, z =  12.4
#> Permutations: 999 (free)
#> 
#> Correlation: Spearman's rank correlation rho
#> Alternative: greater (r is more positive than by chance)
#> Samples: 150

Running tests by name

dist_test() runs any test by name, as dist_ord() does for ordinations. It checks that the variable suits the test (categorical or numeric), and names are matched forgivingly: 'adonis2' finds permanova and 'betadisper' finds permdisp.

dist_test(dm, 'adonis2', 'Species', iris)
#> Permutational multivariate analysis of variance (PERMANOVA) of Bray-Curtis Dissimilarity
#> 
#> F = 532.7, p = 0.001, z = 577.6
#> Permutations: 999 (free)
#> 
#> Model R2 = 0.8788. Model tested as a whole:
#> 
#>           Df SumOfSqs     R2     F Pr(>F)    
#> Model      2   2.3173 0.8788 532.7  0.001 ***
#> Residual 147   0.3197 0.1212                 
#> Total    149   2.6370 1.0000                 
#> ---
#> Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Each result converts to a one-row data frame with the same columns, so several tests combine into one table.

tests <- list_dist_tests(val = 'id', groups = 'categorical')

do.call(rbind, lapply(tests, function (test)
  as.data.frame(dist_test(dm, test, 'Species', iris)) ))
#> Warning: Some squared distances to group centres are negative, and were set to
#> 0.
#>        test stat_name    statistic p_value           z n_perm
#> 1    anosim         R   0.85763011   0.001  79.3963367    999
#> 2      mrpp     delta   0.05794548   0.001 -78.9020283    999
#> 3 permanova         F 532.74262193   0.001 577.5590509    999
#> 4  permdisp         F   0.56763808   0.573  -0.4401028    999

Reproducibility

Permutations come from ecodive’s own random number generator, set by seed (default 0), so the same call always gives the same p-value, on any computer and with any number of cpus. R’s set.seed() has no effect.

identical(
  permanova(dm, 'Species', iris, seed = 1, cpus = 1),
  permanova(dm, 'Species', iris, seed = 1, cpus = 2) )
#> [1] TRUE