The Speedup Test

Sid Touati (1, 2) , Julien Worms (3) , Sébastien Briais (1)

1 PRISM - Parallélisme, Réseaux, Systèmes, Modélisation
2 ALCHEMY - Architectures, Languages and Compilers to Harness the End of Moore Years
3 LMV - Laboratoire de Mathématiques de Versailles

Numerous code optimisation methods are usually experimented by doing multiple observations of the initial and the optimised executions times in order to declare a speedup. Even with fixed input and execution environment, programs executions times vary in general. So hence different kinds of speedups may be reported: the speedup of the average execution time, the speedup of the minimal execution time, the speedup of the median, etc. Many published speedups in the literature are observations of a set of experiments. In order to improve the reproducibility of the experimental results, this technical report presents a rigorous statistical methodology regarding program performance analysis. We rely on well known statistical tests (Shapiro-wilk's test, Fisher's F-test, Student's t-test, Kolmogorov-Smirnov's test, Wilcoxon-Mann-Whitney's test) to study if the observed speedups are statistically significant or not. By fixing $0<\alpha<1$ a desired risk level, we are able to analyse the statistical significance of the average execution time as well as the median. We can also check if $\prob{X>Y}>\frac{1}{2}$, the probability that an individual execution of the optimised code is faster than the individual execution of the initial code. Our methodology defines a consistent improvement compared to the usual performance analysis method in high performance computing as in \cite{Jain:1991:ACS,lilja:book}. We explain in each situation what are the hypothesis that must be checked to declare a correct risk level for the statistics. The Speedup-Test protocol certifying the observed speedups with rigorous statistics is implemented and distributed as an open source tool based on R software.

Mots clés: Code optimisation; program performance evaluation and analysis; statistics

Domaines: Méthodologie [stat.ME]; Performance et fiabilité [cs.PF]; Calcul parallèle, distribué et partagé [cs.DC]

Sid Touati, Julien Worms, Sébastien Briais. The Speedup Test. [Technical Report] Université Versailles Saint-Quentin-En-Yvelines (UVSQ). 2010, pp.38. ⟨inria-00443839v2⟩