The Speedup Test

Sid Touati 1, 2, * Julien Worms 3 Sébastien Briais 1
* Auteur correspondant
2 ALCHEMY - Architectures, Languages and Compilers to Harness the End of Moore Years
LRI - Laboratoire de Recherche en Informatique, UP11 - Université Paris-Sud - Paris 11, Inria Saclay - Ile de France, CNRS - Centre National de la Recherche Scientifique : UMR8623
Abstract : Numerous code optimisation methods are usually experimented by doing multiple observations of the initial and the optimised executions times in order to declare a speedup. Even with fixed input and execution environment, programs executions times vary in general. So hence different kinds of speedups may be reported: the speedup of the average execution time, the speedup of the minimal execution time, the speedup of the median, etc. Many published speedups in the literature are observations of a set of experiments. In order to improve the reproducibility of the experimental results, this technical report presents a rigorous statistical methodology regarding program performance analysis. We rely on well known statistical tests (Shapiro-wilk's test, Fisher's F-test, Student's t-test, Kolmogorov-Smirnov's test, Wilcoxon-Mann-Whitney's test) to study if the observed speedups are statistically significant or not. By fixing $0<\alpha<1$ a desired risk level, we are able to analyse the statistical significance of the average execution time as well as the median. We can also check if $\prob{X>Y}>\frac{1}{2}$, the probability that an individual execution of the optimised code is faster than the individual execution of the initial code. Our methodology defines a consistent improvement compared to the usual performance analysis method in high performance computing as in \cite{Jain:1991:ACS,lilja:book}. We explain in each situation what are the hypothesis that must be checked to declare a correct risk level for the statistics. The Speedup-Test protocol certifying the observed speedups with rigorous statistics is implemented and distributed as an open source tool based on R software.
Liste complète des métadonnées
Contributeur : Sid Touati <>
Soumis le : mardi 26 janvier 2010 - 08:31:41
Dernière modification le : jeudi 9 février 2017 - 15:53:47
Document(s) archivé(s) le : mercredi 30 novembre 2016 - 11:36:03


  • HAL Id : inria-00443839, version 2


Sid Touati, Julien Worms, Sébastien Briais. The Speedup Test. [Technical Report] 2010, pp.38. 〈inria-00443839v2〉



Consultations de
la notice


Téléchargements du document