Principles and Flexibility in Multiple Testing

This e-Closure principle asserts that every such multiple testing method is a special case of a generalized closed testing procedure based on e-values. It applies to a large class of error rates — in particular to the false discovery rate (FDR). We show that any procedure controlling FDR is either a e-Closure  method or is uniformly improved by one. By writing existing methods as special cases of this procedure, we show uniform improvements, as we demonstrate for the Benjamini-Hochberg and the Benjamini-Yekutieli procedures. We also show that methods derived using our novel e-Closure Principle generally control their error rate not just for one rejected set, but simultaneously over many, allowing post hoc flexibility for the researcher. 

This flexibility is important in many application areas including neuroimaging and genomics. Moreover, we show that because all multiple testing methods for all error metrics are derived from the same procedure, researchers may even choose the error metric post hoc, allowing post hoc switching from e.g. familywise error to FDR. Under certain conditions, this flexibility even extends to post hoc choice of the nominal error rate.