Skip to Main content Skip to Navigation

The Speedup Test

Sid Touati 1, 2, * Julien Worms 3 Sébastien Briais 1
* Corresponding author
2 ALCHEMY - Architectures, Languages and Compilers to Harness the End of Moore Years
LRI - Laboratoire de Recherche en Informatique, UP11 - Université Paris-Sud - Paris 11, Inria Saclay - Ile de France, CNRS - Centre National de la Recherche Scientifique : UMR8623
Abstract : Numerous code optimisation methods are usually experimented by doing multiple observations of the initial and the optimised executions times in order to declare a speedup. Even with fixed input and execution environment, programs executions times vary in general. So hence different kinds of speedups may be reported: the speedup of the average execution time, the speedup of the minimal execution time, the speedup of the median, etc. Many published speedups in the literature are observations of a set of experiments. In order to improve the reproducibility of the experimental results, this technical report presents a rigorous statistical methodology regarding program performance analysis. We rely on well known statistical tests (Shapiro-wilk's test, Fisher's F-test, Student's t-test, Kolmogorov-Smirnov's test, Wilcoxon-Mann-Whitney's test) to study if the observed speedups are statistically significant or not. By fixing $0<\alpha<1$ a desired risk level, we are able to analyse the statistical significance of the average execution time as well as the median. We can also check if $\prob{X>Y}>\frac{1}{2}$, the probability that an individual execution of the optimised code is faster than the individual execution of the initial code. Our methodology defines a consistent improvement compared to the usual performance analysis method in high performance computing as in \cite{Jain:1991:ACS,lilja:book}. We explain in each situation what are the hypothesis that must be checked to declare a correct risk level for the statistics. The Speedup-Test protocol certifying the observed speedups with rigorous statistics is implemented and distributed as an open source tool based on R software.
Complete list of metadata
Contributor : Sid Touati <>
Submitted on : Tuesday, January 26, 2010 - 8:31:41 AM
Last modification on : Thursday, July 8, 2021 - 3:48:33 AM
Long-term archiving on: : Wednesday, November 30, 2016 - 11:36:03 AM


  • HAL Id : inria-00443839, version 2


Sid Touati, Julien Worms, Sébastien Briais. The Speedup Test. [Technical Report] 2010, pp.38. ⟨inria-00443839v2⟩



Record views


Files downloads