| Title: | Meta-Analysis of Significance Values |
|---|---|
| Description: | The canonical way to perform meta-analysis involves using effect sizes. When they are not available this package provides a number of methods for meta-analysis of significance values including the methods of Edgington, Fisher, Lancaster, Stouffer, Tippett, and Wilkinson; a number of data-sets to replicate published results; and routines for graphical display. |
| Authors: | Michael Dewey [aut, cre]
|
| Maintainer: | Michael Dewey <[email protected]> |
| License: | GPL-2 |
| Version: | 1.11 |
| Built: | 2025-02-07 03:26:58 UTC |
| Source: | https://github.com/cran/metap |
The canonical way to perform meta-analysis involves using effect sizes. When they are not available this package provides a number of methods for meta-analysis of significance values including the methods of Edgington, Fisher, Lancaster, Stouffer, Tippett, and Wilkinson; a number of data-sets to replicate published results; and routines for graphical display.
Index of help topics:
albatros Albatros plot
allmetap Carry out all or some of the methods in the
package
dat.metap Example data
invchisq Combine p values using the inverse chi squared
method
invt Combine p values using the inverse t method
logitp Combine p values using the logit method
meanp Combine p values by the mean p method
meanz Combine p values using the mean z method
metap-package Meta-Analysis of Significance Values
plotp Q-Q plot of p-values
schweder Schweder and Spjotvoll plot
sumlog Combine p-values by the sum of logs (Fisher's)
method
sump Combine p-values using the sum of p
(Edgington's) method
sumz Combine p-values using the sum of z
(Stouffer's) method
truncated Truncated product methods
two2one Convert two-sided p-values to one-sided
votep Combine p-values by the vote counting method
wilkinsonp Combine p-values using Wilkinson's method
Further information is available in the following vignettes:
compare |
Comparison of methods in the metap package (source, pdf) |
metap |
Introduction to the metap package (source, pdf) |
plotmetap |
Plotting in the metap package (source, pdf) |
Provides a number of ways in which significance levels may be combined in a meta-analysis and includes most ot the methods in Becker (1994). It includes a number of datasets taken from the literature. It also provides a display and an informal graphical test due to Schweder and Spjotvoll (Schweder and Spjotvoll 1982) and the lowest slope line of Benjamini and Hochberg (Benjamini and Hochberg 2000). The albatros plot of Harrison et al (Harrison et al. 2017) is also provided.
Becker BJ (1994).
“Combining significance levels.”
In Cooper H, Hedges LV (eds.), A handbook of research synthesis, 215–230.
Russell Sage, New York.
Benjamini Y, Hochberg Y (2000).
“On the adaptive control of the false discovery rate in multiple testing with independent statistics.”
Journal of Educational and Behavioral Statistics, 25, 60–83.
Harrison S, Jones HE, Martin RM, Lewis SJ, Higgins JPT (2017).
“The albatros plot: A novel graphical tool for presenting the results of diversely reported studies in a systematic review.”
Research Synthesis Methods, 8, 281–289.
Schweder T, Spjotvoll E (1982).
“Plots of –values to evaluate many tests simultaneously.”
Biometrika, 69, 493–502.
The issue of meta-analysis of signficance levels is not completely
unconnected with the topic of adjustment for multiple
comparisons as in for example p.adjust
Produces the albatros plot of Harrison et al
albatros(p, n, axes = list(xlimit = NULL, ylimit = NULL, lefttext = NULL, righttext = NULL), contours = list(type = NULL, contvals = NULL, ltys = NULL, contlabs = NULL), plotpars = list(yscale = NULL, pchs = NULL, cols = NULL), ...)albatros(p, n, axes = list(xlimit = NULL, ylimit = NULL, lefttext = NULL, righttext = NULL), contours = list(type = NULL, contvals = NULL, ltys = NULL, contlabs = NULL), plotpars = list(yscale = NULL, pchs = NULL, cols = NULL), ...)
p |
A vector of \(p\) values |
n |
A vector of sample sizes |
axes |
A list containing
|
contours |
A list containing
|
plotpars |
A list containing
|
... |
Arguments to be passed through to |
Plots \(n\) against \(p\) and draws contours of constant effect size.
The \(p\) value scale on the x-axis is a logarithmic one
but folded at the null value so that the extreme values represent
small probabilities in each direction.
The scale for the y-axis is user selectable.
The original scale in the Stata version is
\((\log_{10}n)^2\) which is obtained by
setting yscale to "classic"
but it is
also possible to use \(\sqrt{n}\)
which is the default or \(n\) by setting
yscale to "n".
If the dataset contains extreme values the
plot may be uninformative and the xlimit
and ylimit parameters my be helpful to clip
the plot.
If they are not specified the function tries
to choose suitable values.
For the contours if the type parameter in contours
is NULL (the default) no contours will be plotted.
The options available are
“smd” (Standardised mean difference)
“corr” (Correlation)
“or” (Odds ratio).
The contours are labelled unless contlabs is set to FALSE.
The function tries to position these so they are not obscured
by the contour lines but if that is unsatisfactory then
it is best to suppress them and position them manually
or use a legend to identify line types.
The pch parameter may either be a single value
or a vector of the same length as p.
It may contain anything which is legal as input to
the parameter of the same name in the graphics
library and will be used for the plotted points.
The cols parameter works similarly.
The values of lefttext and righttext are
used to produce labels under the \(x\)-axis and
are placed level with the label.
By default they are not produced.
The return values from the function may be useful if further annotations are required as they give the axis limits in plotting units.
Returns a list containing
xlims |
The limits for the x-axis in plotting units |
ylims |
The limits for the y-axis in plotting units |
The docontours function is for internal use only
Michael Dewey
Harrison S, Jones HE, Martin RM, Lewis SJ, Higgins JPT (2017). “The albatros plot: A novel graphical tool for presenting the results of diversely reported studies in a systematic review.” Research Synthesis Methods, 8, 281–289.
For plotting parameters
see par
data(dat.metap) validity <- dat.metap$validity fit.v <- albatros(validity$p, validity$n, contours = list(type = "corr", contvals = c(0.25, 0.5, 0.8), ltys = 1:3), axes = list(ylimit = c(1,200), lefttext = "Negative correlation", righttext = "Positive correlation"), main = "Validity") # fit.z <- albatros(dat.metap$zhang$p, dat.metap$zhang$n, contours = list(type = "smd", contvals = c(0.25, 0.5, 1), ltys = 1:3), plotpars = list(pchs = letters[unclass(dat.metap$zhang$phase)]), axes = list(lefttext = "Favours control", righttext = "Favours exercise"), main = "Zhang" )data(dat.metap) validity <- dat.metap$validity fit.v <- albatros(validity$p, validity$n, contours = list(type = "corr", contvals = c(0.25, 0.5, 0.8), ltys = 1:3), axes = list(ylimit = c(1,200), lefttext = "Negative correlation", righttext = "Positive correlation"), main = "Validity") # fit.z <- albatros(dat.metap$zhang$p, dat.metap$zhang$n, contours = list(type = "smd", contvals = c(0.25, 0.5, 1), ltys = 1:3), plotpars = list(pchs = letters[unclass(dat.metap$zhang$phase)]), axes = list(lefttext = "Favours control", righttext = "Favours exercise"), main = "Zhang" )
Enables all or a selected number of the package methods to be executed on the same dataset
allmetap(p, method = NULL, log.p = FALSE) ## S3 method for class 'allmetap' print(x, digits = 5, ...)allmetap(p, method = NULL, log.p = FALSE) ## S3 method for class 'allmetap' print(x, digits = 5, ...)
p |
A vector of \(p\)-values |
method |
A vector of character containing names of methods or ‘ |
log.p |
Logical, are the \(p\)-values passed as logs? |
x |
An object of class ‘ |
digits |
How many digits to print? |
... |
Other arguments to be passed through |
The function uses each of the methods specified
on the data-set in p. If method contains ‘all’
then this over–rides any specific method(s) named.
Returns an object of class
‘allmetap’ and ‘data.frame’ containing columns
p |
The value of p returned |
valid |
The number of valid p values used for that method |
eponym |
The eponym, if any, for the method |
The row names are the names of the methods used.
If any of the methods cannot be applied to that data–set NA are
returned for p and valid.
Michael Dewey
data(dat.metap) beckerp <- dat.metap$beckerp allmetap(beckerp, method = "all") allmetap(beckerp, method = c("sumz", "sumlog"))data(dat.metap) beckerp <- dat.metap$beckerp allmetap(beckerp, method = "all") allmetap(beckerp, method = c("sumz", "sumlog"))
The package contains the following datasets: beckerp,
cholest, edgington, mourning, naep,
rosenthal, teachexpect, validity,
and zhang.
data(dat.metap)data(dat.metap)
A list with the following elements:
beckerpA vector of length 5 hypothetical \(p\) values
cholestA data frame with 34 observations on the following 5 variables.
ntreatA numeric vector of numbers in the treated group
ncontrolA numeric vector of number in the control group
dtreatA numeric vector of number of deaths in the treated group
dcontrolA numeric vector of number of deaths in the control group
pa numeric vector of one sided \(p\) values
edgingtonA vector of length 7 hypothetical \(p\) values
naepA data frame with 34 observations on the following 2 variables.
statea factor with levels AL, AR, AZ, CA, CO, CT, DE, FL, GA, HI, IA, ID, IN, KY, LA, MD, MI, MN, NC, ND, NE, NH, NJ, NM, NY, OH, OK, PA, RI, TX, VA, WI, WV, WY,
pa numeric vector
mourningA data frame with 9 observations on the following 3 variables.
stancea factor with levels No stand, Opponent, Supporter
gradea factor with levels G11-12, G7-8, G9-10
pa numeric vector of \(p\) values
rosenthalA data frame with 5 observations on the following 3 variables.
tA numeric vector of values of \(t\)
dfa numeric vector of degrees of freedom
pa numeric vector of one sided \(p\) values
teachexpectA vector of length 19 hypothetical \(p\) values
validityA data frame with 20 observations on the following 3 variables
nA numeric vector of sample sizes
ra numeric vector of correlation coefficients
pa numeric vector of one sided \(p\) values
zhangA data frame with 22 observations on the following 11 variables
studycharacter, the study names
smdnumeric, the standardised mean difference
lonumeric, the lower confidence limit
hinumeric, the upper confidence limit
ntreatnumeric, the treated sample size
ncontnumeric, the control sample size
nnumeric, the total sample size
phasefactor, what phase the patients were in: acute, healing, healed
sdnumeric, the calculated standard deviation
znumeric, the calculated z
pnumeric, the probability associated with z
beckerpHypothetical \(p\) values from Becker (1994)
cholest\(p\) values from trials of interventions for cholesterol lowering from Sutton et al. (2000)
edgingtonHypothetical \(p\) values from Edgington (1972)
mourningResults from a study of mourning practices of Israeli youth following the assassination of Itzakh Rabin from Benjamini and Hochberg (2000)
naepResults of mathematical achievment scores from the National Assessment of Educational Progress from Benjamini and Hochberg (2000)
rosenthalHypothetical example from Rosenthal (1978)
teachexpect\(p\)-values from studies of the effect of manipulating teacher expectancy on student IQ from Becker (1994)
validityData from studies of validity of student ratings of their instructors from Becker (1994) including correlations and sample sizes as well as \(p\)-values
zhangData from trials of exercise training for patients with cardiovascular disease from Zhang et al. (2016)
The \(p\)-values in cholest have been re-calculated
from other data given in the book
and so are of higher accuracy than the ones
given in the book which are only to two decimal places.
Michael Dewey
Becker BJ (1994).
“Combining significance levels.”
In Cooper H, Hedges LV (eds.), A handbook of research synthesis, 215–230.
Russell Sage, New York.
Benjamini Y, Hochberg Y (2000).
“On the adaptive control of the false discovery rate in multiple testing with independent statistics.”
Journal of Educational and Behavioral Statistics, 25, 60–83.
Edgington ES (1972).
“An additive method for combining probability values from independent experiments.”
Journal of Psychology, 80, 351–363.
Rosenthal R (1978).
“Combining results of independent studies.”
Psychological Bulletin, 85, 185–193.
Sutton AJ, Abrams KR, Jones DR, Sheldon TA, Song F (2000).
Methods for meta-analysis in medical research.
Wiley, Chichester.
Zhang YM, Lu Y, Yang D, Wu HF, Bian ZP, Xu JD, Gu CR, Wang LS, Chen XJ (2016).
“The effects of different initiation time of exercise training on left ventricular remodeling and cardiopulmonary rehabilitation in patients with left ventricular dysfunction after myocardial infarction.”
Disability and Rehabilitation, 38, 268–276.
data(dat.metap)data(dat.metap)
Combine \(p\)-values by the inverse chi-squared method, also known as Lancaster's method
invchisq(p, k, data = NULL, subset = NULL, na.action = na.fail, log.p = FALSE) ## S3 method for class 'invchisq' print(x, ...)invchisq(p, k, data = NULL, subset = NULL, na.action = na.fail, log.p = FALSE) ## S3 method for class 'invchisq' print(x, ...)
p |
A vector of significance values |
k |
A vector of degrees of freedom to use, see details |
data |
Optional data frame containing variables |
subset |
Optional vector of logicals to specify a subset of the \(p\)-values |
na.action |
A function indicating what should happen when data contains NAs |
log.p |
Logical, if TRUE result is returned as log(p) |
x |
An object of class ‘ |
... |
Other arguments to be passed through |
Defined as \[\sum_{i=1}^n \chi^2_{k_i}(p_i) > \chi^2_{\sum{k_i}}(\alpha)\]
If k is a single value it is used for all the p
of which there are n.
If any value of \(k_i\le0\) then the corresponding \(p_i\)
is not included.
The values of \(p_i\) should be such that \(0 < p_i\le 1\) and a warning is given if that is not true. A warning is given if, possibly as a result of removing illegal values, fewer than two values remain and the return values are set to NA.
The plot method for class ‘metap’ calls plotp on the valid p-values.
Inspection of the distribution of \(p\)-values is highly recommended as extreme values in opposite directions do not cancel out. See last example. This may not be what you want.
An object of class ‘invchisq’ and ‘metap’,
a list with entries
chisq |
Value of chi-squared statistic |
df |
Associated degrees of freedom |
p |
Associated p-value |
validp |
The input vector with the illegal values removed |
Michael Dewey
Becker BJ (1994). “Combining significance levels.” In Cooper H, Hedges LV (eds.), A handbook of research synthesis, 215–230. Russell Sage, New York.
Lancaster HO (1949). “Combination of probabilities arising from data in discrete distributions.” Biometrika, 36, 370–382.
data(dat.metap) beckerp <- dat.metap$beckerp invchisq(beckerp, 2) # same as sumlog invchisq(c(0.999, 0.999, 0.001, 0.001), 4) all.equal(exp(invchisq(beckerp, 2, log.p = TRUE)$p), invchisq(beckerp, 2)$p)data(dat.metap) beckerp <- dat.metap$beckerp invchisq(beckerp, 2) # same as sumlog invchisq(c(0.999, 0.999, 0.001, 0.001), 4) all.equal(exp(invchisq(beckerp, 2, log.p = TRUE)$p), invchisq(beckerp, 2)$p)
Combine p values using the inverse t method
invt(p, k, data = NULL, subset = NULL, na.action = na.fail, log.p = FALSE) ## S3 method for class 'invt' print(x, ...)invt(p, k, data = NULL, subset = NULL, na.action = na.fail, log.p = FALSE) ## S3 method for class 'invt' print(x, ...)
p |
A vector of significance values |
k |
A vector of degrees of freedom to use, see details |
data |
Optional data frame containing variables |
subset |
Optional vector of logicals to specify a subset of the \(p\)-values |
na.action |
A function indicating what should happen when data contains NAs |
log.p |
Logical, if TRUE result is returned as log(p) |
x |
An object of class ‘ |
... |
Other arguments to be passed through |
Defined as \[\frac{\sum_{i=1}^n t_{k_i}(p_i)} {\sqrt{\sum_{i=1}^n \frac{k_i}{k_i - 2}}} > z(\alpha) \]
If k is a single value it is used for all the p
of which there are n.
If any value of \(k_i\le2\) then the corresponding \(p_i\)
is not included.
The values of \(p_i\) should be such that \(0 < p_i\le 1\) and a warning is given if that is not true. A warning is given if, possibly as a result of removing illegal values, fewer than two values remain and the return values are set to NA.
The plot method for class ‘metap’ calls plotp on the valid p-values.
An object of class ‘invt’ and ‘metap’,
a list with entries
z |
Value of z |
p |
Associated p-value |
validp |
The input vector with the illegal values removed |
Michael Dewey
Becker BJ (1994). “Combining significance levels.” In Cooper H, Hedges LV (eds.), A handbook of research synthesis, 215–230. Russell Sage, New York.
See also plotp
data(dat.metap) beckerp <- dat.metap$beckerp invt(beckerp, 50) all.equal(exp(invt(beckerp, 50, log.p = TRUE)$p), invt(beckerp, 50)$p)data(dat.metap) beckerp <- dat.metap$beckerp invt(beckerp, 50) all.equal(exp(invt(beckerp, 50, log.p = TRUE)$p), invt(beckerp, 50)$p)
Combine \(p\) values using logit method
logitp(p, log.p = FALSE) ## S3 method for class 'logitp' print(x, ...)logitp(p, log.p = FALSE) ## S3 method for class 'logitp' print(x, ...)
p |
A vector of significance values |
log.p |
Logical, if TRUE result is returned as log(p) |
x |
An object of class ‘ |
... |
Other arguments to be passed through |
Defined as \[t = - \frac{\sum_{i=1}^k \log\frac{p_i}{1 - p_i}}{C}\] where \[C = \sqrt\frac{k \pi^2 (5 k + 2)}{3(5 k + 4)}\] and \(k\) is the number of studies.
The values of \(p_i\) should be such that \(0 < p_i < 1\) and a warning is given if that is not true. A warning is given if, possibly as a result of removing illegal values, fewer than two values remain and the return values are set to NA.
The plot method for class ‘metap’ calls plotp on the valid p-values.
An object of class ‘logitp’ and
‘metap’, a list with entries
t |
Value of Student's \(t\) |
df |
Associated degrees of freedom |
p |
Associated \(p\)-value |
validp |
The input vector with illegal values removed |
Michael Dewey
Becker BJ (1994). “Combining significance levels.” In Cooper H, Hedges LV (eds.), A handbook of research synthesis, 215–230. Russell Sage, New York.
See also plotp
data(dat.metap) teachexpect <- dat.metap$teachexpect logitp(teachexpect) # t = 2.763, df = 99, p = 0.0034, from Becker beckerp <- dat.metap$beckerp logitp(beckerp) # t = 1.62, df = 29, NS, from Becker validity <- dat.metap$validity$p logitp(validity) # t = 9.521, df = 104, p = 3.89 * 10^{-16} all.equal(exp(logitp(validity, log.p = TRUE)$p), logitp(validity)$p)data(dat.metap) teachexpect <- dat.metap$teachexpect logitp(teachexpect) # t = 2.763, df = 99, p = 0.0034, from Becker beckerp <- dat.metap$beckerp logitp(beckerp) # t = 1.62, df = 29, NS, from Becker validity <- dat.metap$validity$p logitp(validity) # t = 9.521, df = 104, p = 3.89 * 10^{-16} all.equal(exp(logitp(validity, log.p = TRUE)$p), logitp(validity)$p)
Combine \(p\) values by the mean p method
meanp(p) ## S3 method for class 'meanp' print(x, ...)meanp(p) ## S3 method for class 'meanp' print(x, ...)
p |
A vector of significance values |
x |
An object of class ‘ |
... |
Other arguments to be passed through |
Defined as \[z = (0.5 - \bar{p}) \sqrt{12k}\] which is a standard normal and due to Edgington (1972)
The values of \(p_i\) should be such that \(0\le p_i\le 1\) and a warning is given if that is not true. A warning is given if, possibly as a result of removing illegal values, fewer than four values remain and the return values are set to NA.
The plot method for class ‘metap’ calls plotp on the valid p-values.
An object of class ‘meanp’ and ‘metap’,
a list with entries
z |
Value of \(z\) |
p |
Associated \(p\)-value |
validp |
The input vector with illegal values removed |
Michael Dewey
Becker BJ (1994).
“Combining significance levels.”
In Cooper H, Hedges LV (eds.), A handbook of research synthesis, 215–230.
Russell Sage, New York.
Edgington ES (1972).
“A normal curve method for combining probability values from independent experiments.”
Journal of Psychology, 82, 85–89.
Rosenthal R (1978).
“Combining results of independent studies.”
Psychological Bulletin, 85, 185–193.
See also plotp
data(dat.metap) rosenthal <- dat.metap$rosenthal meanp(rosenthal$p) # 2.17, p = 0.015 one taileddata(dat.metap) rosenthal <- dat.metap$rosenthal meanp(rosenthal$p) # 2.17, p = 0.015 one tailed
Combines p values using the mean of z method
meanz(p, log.p = FALSE) ## S3 method for class 'meanz' print(x, ...)meanz(p, log.p = FALSE) ## S3 method for class 'meanz' print(x, ...)
p |
A vector of significance values |
log.p |
Logical, if TRUE result is returned as log(p) |
x |
An object of class ‘ |
... |
Other arguments to be passed through |
Let \[\bar{z} = \sum_{i=1}^k \frac{z(p_i)}{k}\] and \[s_{\bar{z}} = \frac{s_z}{\sqrt{k}}\] Defined as \[ \frac{\bar{z}}{s_{\bar{z}}} > t_{k-1}(\alpha) \]
The values of \(p_i\) should be such that \(0\le p_i\le 1\) and a warning is given if that is not true. A warning is given if, possibly as a result of removing illegal values, fewer than two values remain and the return values are set to NA. As can be seen if all the \(p_i\) are equal or close to equal this gives a \(t=\pm\infty\) leading to a returned value of 0 or 1. A set of \(p\) values with small variance will necessarily give a large value for \[\frac{\bar{z}}{s_{\bar{z}}}\] and hence a small \(p\) value which may be smaller than that for another set all of whose primary values are less than any in the first set. See examples for a demonstration.
The plot method for class ‘metap’ calls plotp on the valid p-values.
An object of class ‘meanz’ and
‘metap’, a list with entries
z |
The value of the mean \(z\) statistic |
p |
The associated \(p\) value |
validp |
The input vector with illegal values removed |
Michael Dewey
Becker BJ (1994). “Combining significance levels.” In Cooper H, Hedges LV (eds.), A handbook of research synthesis, 215–230. Russell Sage, New York.
See also plotp
data(dat.metap) beckerp <- dat.metap$beckerp meanz(beckerp) meanz(c(0.1, 0.2)) # greater than next example meanz(c(0.3, 0.31)) # less than above all.equal(exp(meanz(beckerp, log.p = TRUE)$p), meanz(beckerp)$p)data(dat.metap) beckerp <- dat.metap$beckerp meanz(beckerp) meanz(c(0.1, 0.2)) # greater than next example meanz(c(0.3, 0.31)) # less than above all.equal(exp(meanz(beckerp, log.p = TRUE)$p), meanz(beckerp)$p)
Produces a Q–Q plot of the p-values supplied
plotp(pvals, plotversion = "qqconf", ...)plotp(pvals, plotversion = "qqconf", ...)
pvals |
A vector of \(p\)–values |
plotversion |
Whether to use the qqconf version or the original |
... |
Other parameters to pass through to |
After eliminating out of range \(p\)–values
and missing values produces a Q–Q plot.
If the package qqconf is unavailable or
plotversion is set other than "qqconf"
then the plot uses the function from base graphics.
The line shown is
through the quantiles (by default the 0.25 and 0.75).
If the package qqconf is available then the plot
uses that package's function and as well as the Q–Q plot
produces a confidence region.
Parameters can be passed to qq_conf_plot to
exert further control over the plot.
If plotversion is set to "qqconf"
but package qqconf is not available
then a warning is given and no plot is produced.
Invisibly returns a list consisting of
validp |
The valid \(p\)–values |
Michael Dewey
qqplot and qqline for
the original plot and qq_conf_plot for
the new version.
data(dat.metap) cholest <- dat.metap$cholest$p plotp(cholest)data(dat.metap) cholest <- dat.metap$cholest$p plotp(cholest)
Produces the plot suggested by Schweder and Spjotvoll to display a collection of \(p\)-values and also optionally draws the lowest slope line suggested by Benjamini and Hochberg
schweder(p, xlab = "Rank of p", ylab = "p", drawline = NULL, bh.lwd = 1, bh.lty = "solid", bh.col = "black", ls.control = list(frac = NULL), ls.lwd = 1, ls.lty = "dotted", ls.col = "black", ab.control = list(a = NULL, b = NULL), ab.lwd = 1, ab.lty = "dashed", ab.col = "black", ...)schweder(p, xlab = "Rank of p", ylab = "p", drawline = NULL, bh.lwd = 1, bh.lty = "solid", bh.col = "black", ls.control = list(frac = NULL), ls.lwd = 1, ls.lty = "dotted", ls.col = "black", ab.control = list(a = NULL, b = NULL), ab.lwd = 1, ab.lty = "dashed", ab.col = "black", ...)
p |
A vector of \(p\)-values |
xlab |
Label for plot \(x\)-axis |
ylab |
Label for plot \(y\)-axis |
drawline |
Which line, if any, to draw. See Details |
bh.lwd |
Width of the lowest slope line |
bh.lty |
Line type of the lowest slope line |
bh.col |
Colour of the lowest slope line |
ls.control |
A list containing a named element |
ls.lwd |
Width of the least squares line |
ls.lty |
Line type of the least squares line |
ls.col |
Colour of the least squares line |
ab.control |
A list containing two elements |
ab.lwd |
Width of the specified line |
ab.lty |
Line type of the specified line |
ab.col |
Colour of the specified line |
... |
Further parameters to be passed through to plot |
After removing invalid \(p\)-values plots them.
Any graphics parameters passed in ... affect this part
of the plot only.
Note that the axes are chosen according to the scheme of Banjamini and
Hochberg, Schweder and Spjotvoll used a different system.
By setting drawline appropriately up to three
lines may be drawn.
If drawline includes the string "bh"
the lowest slope line of Benjaimin and Hochberg is drawn.
No further parameters are needed here but the characteristics of the
line may be set: width, line type, and colour.
If drawline includes the string "ls"
a least squares line is drawn passing through the point
\(k+1, 1\).
The parameter frac specified what fraction of the
values be used for this and may need experimentation
to obtain a suitable line.
The characteristics of the
line may be set: width, line type, and colour.
If drawline includes the string "ab"
a user specified line is drawn.
The parameters a and b specify the intercept
and slope.
The characteristics of the
line may be set: width, line type, and colour.
A list containing
p |
The values of \(p\) plotted |
bh.params |
A list containing at least items |
ls.params |
A list containing at least items |
ab.params |
A list containing at least items |
Michael Dewey
Becker BJ (1994). “Combining significance levels.” In Cooper H, Hedges LV (eds.), A handbook of research synthesis, 215–230. Russell Sage, New York.
Benjamini Y, Hochberg Y (2000). “On the adaptive control of the false discovery rate in multiple testing with independent statistics.” Journal of Educational and Behavioral Statistics, 25, 60–83.
Schweder T, Spjotvoll E (1982). “Plots of P–values to evaluate many tests simultaneously.” Biometrika, 69, 493–502.
data(dat.metap) teachexpect <- dat.metap$teachexpect schweder(teachexpect)data(dat.metap) teachexpect <- dat.metap$teachexpect schweder(teachexpect)
Combine \(p\)-values by the sum of logs method, also known as Fisher's method, and sometimes as the chi-square (2) method
sumlog(p, log.p = FALSE, log.input = FALSE) ## S3 method for class 'sumlog' print(x, ...)sumlog(p, log.p = FALSE, log.input = FALSE) ## S3 method for class 'sumlog' print(x, ...)
p |
A vector of significance values |
log.p |
Logical, if TRUE result is returned as log(p) |
log.input |
Logical, if TRUE the input \(p\) values are assumed to be logged |
x |
An object of class ‘ |
... |
Other arguments to be passed through |
The method relies on the fact that \[\sum_{i=1}^k - 2 \log p_i\] is a chi-squared with \(2 k\) df where \(k\) is the number of studies (Fisher 1925).
The values of \(p_i\) should be such that \(0 < p_i\le 1\) and a warning is given if that is not true. A warning is given if, possibly as a result of removing illegal values, fewer than two values remain and the return values are set to NA.
The log.input parameter may be beneficial
when the input values are already logged and would be
small if exponentiated since it avoids a
conversion.
The plot method for class ‘metap’ calls plotp on the valid p-values.
Inspection of the distribution of \(p\)-values is highly recommended as extreme values in opposite directions do not cancel out. See last example. This may not be what you want.
An object of class ‘sumlog’ and ‘metap’,
a list with entries
chisq |
Value of chi-squared statistic |
df |
Associated degrees of freedom |
p |
Associated \(p\)-value |
validp |
The input vector with the illegal values removed |
Michael Dewey
Becker BJ (1994).
“Combining significance levels.”
In Cooper H, Hedges LV (eds.), A handbook of research synthesis, 215–230.
Russell Sage, New York.
Fisher RA (1925).
Statistical methods for research workers.
Oliver and Boyd, Edinburgh.
Rosenthal R (1978).
“Combining results of independent studies.”
Psychological Bulletin, 85, 185–193.
Sutton AJ, Abrams KR, Jones DR, Sheldon TA, Song F (2000).
Methods for meta-analysis in medical research.
Wiley, Chichester.
See also plotp
data(dat.metap) teachexpect <- dat.metap$teachexpect sumlog(teachexpect) # chisq = 69.473, df = 38, p = 0.0014, from Becker beckerp <- dat.metap$beckerp sumlog(beckerp) # chisq = 18.533, df = 10, sig rosenthal <- dat.metap$rosenthal sumlog(rosenthal$p) # chisq = 22.97, df = 10, p = 0.006 one sided cholest <- dat.metap$cholest$p sumlog(cholest) # chisq = 58.62, df = 68, p = 0.78 validity <- dat.metap$validity$p sumlog(validity) # chisq = 159.82, df = 40, p = 2.91 * 10^{-16} sumlog(c(0.0001, 0.0001, 0.9999, 0.9999)) # is significant all.equal(exp(sumlog(validity, log.p = TRUE)$p), sumlog(validity)$p) all.equal(sumlog(log(validity), log.input = TRUE)$p, sumlog(validity)$p)data(dat.metap) teachexpect <- dat.metap$teachexpect sumlog(teachexpect) # chisq = 69.473, df = 38, p = 0.0014, from Becker beckerp <- dat.metap$beckerp sumlog(beckerp) # chisq = 18.533, df = 10, sig rosenthal <- dat.metap$rosenthal sumlog(rosenthal$p) # chisq = 22.97, df = 10, p = 0.006 one sided cholest <- dat.metap$cholest$p sumlog(cholest) # chisq = 58.62, df = 68, p = 0.78 validity <- dat.metap$validity$p sumlog(validity) # chisq = 159.82, df = 40, p = 2.91 * 10^{-16} sumlog(c(0.0001, 0.0001, 0.9999, 0.9999)) # is significant all.equal(exp(sumlog(validity, log.p = TRUE)$p), sumlog(validity)$p) all.equal(sumlog(log(validity), log.input = TRUE)$p, sumlog(validity)$p)
Combine \(p\)-values using the sum p method
sump(p) ## S3 method for class 'sump' print(x, ...)sump(p) ## S3 method for class 'sump' print(x, ...)
p |
A vector of significance values |
x |
An object of class ‘ |
... |
Other arguments to be passed through |
Defined as \[\frac{(\sum_{i=1}^k p_i)^k}{k!} - {k \choose 1}\frac{(\sum_{i=1}^k p_i - 1)^k}{k!} + {k \choose 2}\frac{(\sum_{i=1}^k p_i - 2)^k}{k!} ...\] where there are \(k\) studies and the series continues until the numerator becomes negative (Edgington 1972).
Some authors use a simpler version
\[\frac{(\sum_{i=1}^k p_i)^k}{k!}\]
but this can be very conservative when
\(\sum_{i=1}^k p_i > 1\).
There seems no particular need to use this method but
it is returned as the value of conservativep
for use in checking published values.
The values of \(p_i\) should be such that \(0\le p_i\le 1\) and a warning is given if that is not true. A warning is given if, possibly as a result of removing illegal values, fewer than two values remain and the return values are set to NA. A warning is given when the internal calculations are likely to have been subject to numerical error and an alternative method should be used to check the result.
The plot method for class ‘metap’ calls plotp on the valid p-values.
An object of class ‘sump’ and
‘metap’, a list with entries
p |
The transformed sum of the \(p\)-values |
conservativep |
See details |
validp |
The input vector with illegal values removed |
Michael Dewey
Edgington ES (1972). “An additive method for combining probability values from independent experiments.” Journal of Psychology, 80, 351–363.
See also plotp
data(dat.metap) edgington <- dat.metap$edgington sump(edgington) # p = 0.097data(dat.metap) edgington <- dat.metap$edgington sump(edgington) # p = 0.097
Combine \(p\)-values using the sum of z method
sumz(p, weights = NULL, data = NULL, subset = NULL, na.action = na.fail, log.p = FALSE, log.input = FALSE) ## S3 method for class 'sumz' print(x, ...)sumz(p, weights = NULL, data = NULL, subset = NULL, na.action = na.fail, log.p = FALSE, log.input = FALSE) ## S3 method for class 'sumz' print(x, ...)
p |
A vector of significance values |
weights |
A vector of weights |
data |
Optional data frame containing variables |
subset |
Optional vector of logicals to specify a subset of the \(p\)-values |
na.action |
A function indicating what should happen when data contains NAs |
log.p |
Logical, if TRUE result is returned as log(p) |
log.input |
Logical, if TRUE the input \(p\) values are assumed to be logged |
x |
An object of class ‘ |
... |
Other arguments to be passed through |
Defined as \[\frac{\sum_{i=1}^k w_i z(p_i)}{\sqrt {\sum_{i=1}^k w_i ^ 2}}\] is a \(z\) where \(k\) is the number of studies and \(w\) are the weights (Stouffer et al. 1949). By default the weights are equal. In the absence of effect sizes (in which case a method for combining effect sizes would be more appropriate anyway) best results are believed to be obtained with weights proportional to the square root of the sample sizes (Zaykin 2011)
The values of \(p_i\) should be such that \(0 < p_i < 1\) and a warning is given if that is not true. A warning is given if, possibly as a result of removing illegal values, fewer than two values remain and the return values are set to NA. If the omitted \(p\) values had supplied weights a further warning is issued.
The log.input parameter may be beneficial
when the input values are already logged and would be
small if exponentiated since it avoids a
conversion.
The plot method for class ‘metap’ calls plotp on the valid p-values.
An object of class ‘sumz’ and
‘metap’, a list with entries
z |
Transformed sum of \(z\) values |
p |
Associated \(p\)-value |
validp |
The input vector with illegal values removed |
weights |
The weight vector corresponding to |
Michael Dewey
Becker BJ (1994).
“Combining significance levels.”
In Cooper H, Hedges LV (eds.), A handbook of research synthesis, 215–230.
Russell Sage, New York.
Rosenthal R (1978).
“Combining results of independent studies.”
Psychological Bulletin, 85, 185–193.
Stouffer SA, Suchman EA, DeVinney LC, Star SA, Williams RMJ (1949).
The American soldier, vol 1: Adjustment during army life.
Princeton University Press, Princeton.
Zaykin DV (2011).
“Optimally weighted Z–test is a powerful method for combining probabilities in meta–analysis.”
Journal of Evolutionary Biology, 24, 1836–1841.
See also plotp
data(dat.metap) teachexpect <- dat.metap$teachexpect sumz(teachexpect) # z = 2.435, p = 0.0074, from Becker beckerp <- dat.metap$beckerp sumz(beckerp) # z = 1.53, NS, from Beckerp rosenthal <- dat.metap$rosenthal sumz(rosenthal$p) # 2.39, p = 0.009 sumz(p, df, rosenthal) # 3.01, p = 0.0013 validity <- dat.metap$validity$p sumz(validity) # z = 8.191, p = 1.25 * 10^{-16} all.equal(exp(sumz(validity, log.p = TRUE)$p), sumz(validity)$p) all.equal(sumz(log(validity), log.input = TRUE)$p, sumz(validity)$p)data(dat.metap) teachexpect <- dat.metap$teachexpect sumz(teachexpect) # z = 2.435, p = 0.0074, from Becker beckerp <- dat.metap$beckerp sumz(beckerp) # z = 1.53, NS, from Beckerp rosenthal <- dat.metap$rosenthal sumz(rosenthal$p) # 2.39, p = 0.009 sumz(p, df, rosenthal) # 3.01, p = 0.0013 validity <- dat.metap$validity$p sumz(validity) # z = 8.191, p = 1.25 * 10^{-16} all.equal(exp(sumz(validity, log.p = TRUE)$p), sumz(validity)$p) all.equal(sumz(log(validity), log.input = TRUE)$p, sumz(validity)$p)
Implements two procedures for truncated versions of Fisher's method
truncated(p, ptrunc = NULL, rtrunc = NULL, ...) ## S3 method for class 'truncated' print(x, ...) ## S3 method for class 'truncated' summary(object, ...) ## S3 method for class 'truncated' plot(x, pparams = list(pchs = c(16, 1), pcols = c("black", "black")), ...)truncated(p, ptrunc = NULL, rtrunc = NULL, ...) ## S3 method for class 'truncated' print(x, ...) ## S3 method for class 'truncated' summary(object, ...) ## S3 method for class 'truncated' plot(x, pparams = list(pchs = c(16, 1), pcols = c("black", "black")), ...)
p |
A vector of significance values |
ptrunc |
Value of \(p\) to truncate at. See Details |
rtrunc |
Which rank \(p\) to truncate at. See Details |
x |
An object of class ‘ |
object |
An object of class ‘ |
... |
Other arguments to be passed through |
pparams |
A list containing a vector |
Uses routines from TFisher to implement truncated Fisher (Zaykin et al. 2007; Zhang et al. 2018) and from mutoss to implement rank-truncated Fisher (Dudbridge and Koeleman 2003)
The print method prints the statistic and its associated
\(p\)-value. The summary method also prints how many
of the input values were actually used. The plot method
provides an index plot of the \(p\)-values.
By default used \(p\)-values are plotted
with a black filled circle and those not used with a black
unfilled circle. The colours and plotting symbols can be changed
in pparams. The first element of each vector is for the
used \(p\)-values and the second for the unused ones.
An object of class ‘truncated’
a list with entries
p |
The resultant overall \(p\)-value |
stat |
The test statistic |
validp |
The input vector with illegal values removed |
ptrunc |
The \(p\)-value used for truncation |
trunc |
The rank value used for truncation |
n |
A vector containing |
method |
The method used |
Only one of ptrunc or rtrunc is returned
as appropriate. If the function encounters an error
it returns NA for the value of p and gives
a warning message.
The effect of specifying both ptrunc
and rtrunc is undefined.
Michael Dewey
Dudbridge F, Koeleman BPC (2003).
“Rank truncated product of P–values, with application to genomewide association scans.”
Genetic Epidemiology, 25, 360–366.
Zaykin DV, Zhivotovsky LA, Czika W, Shao S, Wolfinger RD (2007).
“Combining p–values in large–scale genomics experiments.”
Pharmaceutical Statistics, 6, 217–236.
Zhang H, Tong T, Landers J, Wu Z (2018).
“TFisher tests: optimal and adaptive thresholding for combining p–values.”
arXiv.
https://arxiv.org/abs/1801.04309.
ranktruncated, stat.tpm,
and p.tpm
data(dat.metap) validity <- dat.metap$validity truncated(validity, ptrunc = 0.5) truncated(validity, rtrunc = 5)data(dat.metap) validity <- dat.metap$validity truncated(validity, ptrunc = 0.5) truncated(validity, rtrunc = 5)
Convert two-sided p-values to one-sided
two2one(p, two = NULL, invert = NULL)two2one(p, two = NULL, invert = NULL)
p |
A vector of p-values |
two |
A logical vector defining which \(p\)-values are two-sided and to be converted |
invert |
A logical vector defining which \(p\)-values are to be inverted |
The \(p\)-values supplied to the other functions should be
one-sided and all in the same direction.
This convenience function will convert two-sided to
one-sided and invert as necessary.
By default it is assumed that all \(p\)-values are two-sided
and the function converts them all to one-sided.
Use invert to specify if some of the tests
resulted in outcomes in the other direction.
So for example a value of 0.05 will be converted to 0.025 unless
invert is TRUE in which case it is converted to 0.975.
A vector of one-sided \(p\)-values all in the correct direction
Michael Dewey
data(dat.metap) rosenthal <- dat.metap$rosenthal twop <- with(rosenthal, (pt(t, df))) two2one(twop, two = rep(FALSE, 5), invert = rep(TRUE, 5)) # restore to one-sideddata(dat.metap) rosenthal <- dat.metap$rosenthal twop <- with(rosenthal, (pt(t, df))) two2one(twop, two = rep(FALSE, 5), invert = rep(TRUE, 5)) # restore to one-sided
Combine \(p\)-values by the vote counting method
votep(p, alpha = 0.5) ## S3 method for class 'votep' print(x, ...)votep(p, alpha = 0.5) ## S3 method for class 'votep' print(x, ...)
p |
A vector of significance values |
alpha |
A value defining the significance value limits, see Details |
x |
An object of class ‘ |
... |
Other arguments to be passed through |
By default splits the \(p\)-values at 0.5 with those below counting as positive and those above counting as negative. However setting alpha allows specifying a neutral zone. In that case values between alpha and 1 - alpha will be assumed to be neutral and only those outside the zone counted as positive or negative. If the value of alpha is greater than unity it is assumed to be a percentage.
The values of \(p_i\) should be such that \(0 < p_i < 1\) and a warning is given if that is not true. A warning is given if, possibly as a result of removing illegal values, fewer than two values remain and the return values are set to NA.
The plot method for class ‘metap’ calls plotp on the valid p-values.
An object of class ‘votep’
and ‘votep’, a list with entries
p |
\(p\) value |
pos |
Number of positives |
neg |
Number of negatives |
alpha |
The cut–off significance value lying in the range 0 to 1 |
validp |
The input vector with illegal values removed |
Michael Dewey
Becker BJ (1994). “Combining significance levels.” In Cooper H, Hedges LV (eds.), A handbook of research synthesis, 215–230. Russell Sage, New York.
See also plotp
data(dat.metap) beckerp <- dat.metap$beckerp votep(beckerp)data(dat.metap) beckerp <- dat.metap$beckerp votep(beckerp)
Combine \(p\)-values using Wilkinson's method
wilkinsonp(p, r = 1, alpha = 0.05, log.p = FALSE) maximump(p, alpha = 0.05, log.p = FALSE) minimump(p, alpha = 0.05, log.p = FALSE) ## S3 method for class 'wilkinsonp' print(x, ...) ## S3 method for class 'maximump' print(x, ...) ## S3 method for class 'minimump' print(x, ...)wilkinsonp(p, r = 1, alpha = 0.05, log.p = FALSE) maximump(p, alpha = 0.05, log.p = FALSE) minimump(p, alpha = 0.05, log.p = FALSE) ## S3 method for class 'wilkinsonp' print(x, ...) ## S3 method for class 'maximump' print(x, ...) ## S3 method for class 'minimump' print(x, ...)
p |
A vector of significance values |
r |
Use the \(r\)th smallest \(p\) value |
alpha |
The significance level |
log.p |
Logical, if TRUE result is returned as log(p) |
x |
An object of class ‘ |
... |
Other arguments to be passed through |
Wilkinson (Wilkinson 1951)
originally proposed his method in the context of
simultaneous statistical inference: the probability
of obtaining \(r\) or more significant statistics by
chance in a group of \(k\).
The values are obtained from the Beta distribution, see
pbeta.
If alpha is greater than unity
it is assumed to be a percentage. Either values greater than 0.5 (assumed to
be confidence coefficient) or less than 0.5 are accepted.
The values of \(p_i\) should be such that \(0\le p_i\le 1\) and a warning is given if that is not true. A warning is given if, possibly as a result of removing illegal values, fewer than two values remain and the return values are set to NA.
maximump and
minimump each provide a wrapper for wilkinsonp
for the special case when \(r = \mathrm{length}(p)\)
or \(r=1\) respectively and each has its own
print method.
The method of minimum \(p\) is also known as Tippett's method
(Tippett 1931).
The plot method for class ‘metap’ calls plotp on the valid p-values.
Inspection of the distribution of \(p\)-values is highly recommended as extreme values in opposite directions do not cancel out. See last example. This may not be what you want.
An object of class ‘wilkinsonp’
and ‘metap’ or of class ‘maximump’
and ‘metap’ or of class ‘minimump’
and ‘metap’,
a list with entries
p |
The \(p\)-value resulting from the meta–analysis |
pr |
The \(r\)th smallest \(p\) value used |
r |
The value of \(r\) |
critp |
The critical value at which the \(r\)th value
would have been significant for the chosen |
validp |
The input vector with illegal values removed |
The value of critp is always on the raw scale even
if log.p has been set to TRUE
Michael Dewey
Becker BJ (1994).
“Combining significance levels.”
In Cooper H, Hedges LV (eds.), A handbook of research synthesis, 215–230.
Russell Sage, New York.
Birnbaum A (1954).
“Combining independent tests of significance.”
Journal of the American Statistical Association, 49, 559–574.
Tippett LHC (1931).
The methods of statistics.
Williams and Norgate, London.
Wilkinson B (1951).
“A statistical consideration in psychological research.”
Psychological Bulletin, 48, 156–158.
See also plotp
data(dat.metap) beckerp <- dat.metap$beckerp minimump(beckerp) # signif = FALSE, critp = 0.0102, minp = 0.016 teachexpect <- dat.metap$teachexpect minimump(teachexpect) # crit 0.0207, note Becker says minp = 0.0011 wilkinsonp(c(0.223, 0.223), r = 2) # Birnbaum, just signif validity <- dat.metap$validity$p minimump(validity) # minp = 0.00001, critp = 1.99 * 10^{-4} minimump(c(0.0001, 0.0001, 0.9999, 0.9999)) # is significant all.equal(exp(minimump(validity, log.p = TRUE)$p), minimump(validity)$p) all.equal(exp(maximump(validity, log.p = TRUE)$p), maximump(validity)$p)data(dat.metap) beckerp <- dat.metap$beckerp minimump(beckerp) # signif = FALSE, critp = 0.0102, minp = 0.016 teachexpect <- dat.metap$teachexpect minimump(teachexpect) # crit 0.0207, note Becker says minp = 0.0011 wilkinsonp(c(0.223, 0.223), r = 2) # Birnbaum, just signif validity <- dat.metap$validity$p minimump(validity) # minp = 0.00001, critp = 1.99 * 10^{-4} minimump(c(0.0001, 0.0001, 0.9999, 0.9999)) # is significant all.equal(exp(minimump(validity, log.p = TRUE)$p), minimump(validity)$p) all.equal(exp(maximump(validity, log.p = TRUE)$p), maximump(validity)$p)