The Speedup Test

Sid Touati 1, 2, * Julien Worms 3 Sébastien Briais 1
* Auteur correspondant
2 ALCHEMY - Architectures, Languages and Compilers to Harness the End of Moore Years
LRI - Laboratoire de Recherche en Informatique, UP11 - Université Paris-Sud - Paris 11, Inria Saclay - Ile de France, CNRS - Centre National de la Recherche Scientifique : UMR8623
Abstract : Numerous code optimisation methods are usually experimented by doing multiple observations of the initial and the optimised executions times in order to declare a speedup. Even with fixed input and execution environment, programs executions times vary in general. So hence different kinds of speedups may be reported: the speedup of the average execution time, the speedup of the minimal execution time, the speedup of the median, etc. Many published speedups in the literature are observations of a set of experiments. In order to improve the reproducibility of the experimental results, this technical report presents a rigorous statistical methodology regarding program performance analysis. We rely on well known statistical tests (Shapiro-wilk's test, Fisher's F-test, Student's t-test, Kolmogorov-Smirnov's test, Wilcoxon-Mann-Whitney's test) to study if the observed speedups are statistically significant or not. By fixing $0<\alpha<1$ a desired risk level, we are able to analyse the statistical significance of the average execution time as well as the median. We can also check if $\prob{X>Y}>\frac{1}{2}$, the probability that an individual execution of the optimised code is faster than the individual execution of the initial code. Our methodology defines a consistent improvement compared to the usual performance analysis method in high performance computing as in \cite{Jain:1991:ACS,lilja:book}. We explain in each situation what are the hypothesis that must be checked to declare a correct risk level for the statistics. The Speedup-Test protocol certifying the observed speedups with rigorous statistics is implemented and distributed as an open source tool based on R software.
Liste complète des métadonnées

https://hal.inria.fr/inria-00443839
Contributeur : Sid Touati <>
Soumis le : jeudi 7 janvier 2010 - 14:12:57
Dernière modification le : vendredi 6 février 2015 - 13:43:43
Document(s) archivé(s) le : jeudi 18 octobre 2012 - 11:40:36

Identifiants

  • HAL Id : inria-00443839, version 1

Citation

Sid Touati, Julien Worms, Sébastien Briais. The Speedup Test. [Technical Report] 2010, pp.35. <inria-00443839v1>

Partager

Métriques

Consultations de
la notice

40

Téléchargements du document

136