Skip to Main content Skip to Navigation
Conference papers

Bayesian performance analysis for black-box optimization benchmarking

Abstract : The most commonly used statistics in Evolutionary Computation (EC) are of the Wilcoxon-Mann-Whitney-test type, in its either paired or non-paired version. However, using such statistics for drawing performance comparisons has several known drawbacks. At the same time, Bayesian inference for performance analysis is an emerging statistical tool, which has the potential to become a promising complement to the statistical perspectives offered by the aforementioned p-value type test. This work exhibits the practical use of Bayesian inference in a typical EC setting, where several algorithms are to be compared with respect to various performance indicators. Explicitly we examine performance data of 11 evolutionary algorithms (EAs) over a set of 23 discrete optimization problems in several dimensions. Using this data, and following a brief introduction to the relevant Bayesian inference practice, we demonstrate how to draw the algorithms' probabilities of winning. Apart from fixed-target and fixed-budget results for the individual problems, we also provide an illustrative example per groups of problems. We elaborate on the computational steps, explain the associated uncertainties, and articulate considerations such as the prior distribution and the sample sizing. We also present as a reference the classical p-value tests.
Document type :
Conference papers
Complete list of metadatas

Cited literature [27 references]  Display  Hide  Download

https://hal.sorbonne-universite.fr/hal-02179609
Contributor : Carola Doerr <>
Submitted on : Tuesday, January 14, 2020 - 3:32:19 PM
Last modification on : Monday, January 20, 2020 - 10:27:58 AM
Long-term archiving on: : Wednesday, April 15, 2020 - 6:36:15 PM

File

bl Calvo EtAl GECCO 19 Bayesia...
Files produced by the author(s)

Licence


Distributed under a Creative Commons Attribution 4.0 International License

Identifiers

Citation

Borja Calvo, Ofer Shir, Josu Ceberio, Carola Doerr, Hao Wang, et al.. Bayesian performance analysis for black-box optimization benchmarking. Genetic and Evolutionary Computation Conference GECCO 2019, Jul 2019, Prague, Czech Republic. pp.1789-1797, ⟨10.1145/3319619.3326888⟩. ⟨hal-02179609⟩

Share

Metrics

Record views

98

Files downloads

214