Making a case for (Hyper-)parameter tuning as benchmark problems

Abstract : One of the biggest challenges in evolutionary computation concerns the selection and configuration of a best-suitable heuristic for a given problem. While in the past both of these problems have primarily been addressed by building on experts' experience, the last decade has witnessed a significant shift towards automated decision making, which capitalizes on techniques proposed in the machine learning literature. A key success factor in automated algorithm selection and configuration are good training sets, whose performance data can be leveraged to build accurate performance prediction models. With the long-term goal to build landscape-aware parameter control mechanisms for iterative optimization heuristics, we consider in this discussion paper the question how well the 24 functions from the BBOB test bed cover the characteristics of (hyper-)parameter tuning problems. To this end, we perform a preliminary landscape analysis of two hyper-parameter selection problems, and compare their feature values with those of the BBOB functions. While we do see a good fit for one of the tuning problems, our findings also indicate that some parameter tuning problems might not be very well represented by the BBOB functions. This raises the question if one can nevertheless deduce reliable performance-prediction models for hyper-parameter tuning problems from the BBOB test bed, or whether for this specific target the BBOB benchmark should be adjusted, by adding or replacing some of its functions. Independently of the aspect of training automated algorithm selection and configuration techniques, hyper-parameter tuning problems offer a plethora of problems which might be worthwhile to study in the context of benchmarking iterative optimization heuristics.
Document type :
Conference papers
Complete list of metadatas

https://hal.sorbonne-universite.fr/hal-02179587
Contributor : Carola Doerr <>
Submitted on : Wednesday, July 10, 2019 - 11:12:42 PM
Last modification on : Friday, July 12, 2019 - 10:07:43 AM

Identifiers

Citation

Carola Doerr, Johann Dréo, Pascal Kerschke. Making a case for (Hyper-)parameter tuning as benchmark problems. Genetic and Evolutionary Computation Conference, Companion Material, Jul 2019, Prague, Czech Republic. pp.1755-1764, ⟨10.1145/3319619.3326857⟩. ⟨hal-02179587⟩

Share

Metrics

Record views

20