Finite Sample Improvement of Akaike's Information Criterion

Adrien Saumard (1, 2) , Fabien Navarro (3)

1 CREST - Centre de Recherche en Economie et Statistique [Bruz]
2 ENSAI - Ecole Nationale de la Statistique et de l'Analyse de l'Information [Bruz]
3 SAMM - Statistique, Analyse et Modélisation Multidisciplinaire (SAmos-Marin Mersenne)

Considering the selection of frequency histograms, we propose a modification of Akaike's Information Criterion that avoids overfitting, even when the sample size is small. We call this correction an over-penalization procedure. We emphasize that the principle of unbiased risk estimation for model selection can indeed be improved by addressing excess risk deviations in the design of the penalization procedure. On the theoretical side, we prove sharp oracle inequalities for the Kullback-Leibler divergence. These inequalities are valid with positive probability for any sample size and include the estimation of unbounded log-densities. Along the proofs, we derive several analytical lemmas related to the Kullback-Leibler divergence, as well as concentration inequalities, that are of independent interest. In a simulation study, we also demonstrate state-of-theart performance of our over-penalization criterion for bin size selection, in particular outperforming AICc procedure.

Mots clés: model selection; bin size; AIC corrected; overpenalization; small sample size

Domaines: Statistiques [math.ST]; Apprentissage [cs.LG]; Machine Learning [stat.ML]

Adrien Saumard, Fabien Navarro. Finite Sample Improvement of Akaike's Information Criterion. IEEE Transactions on Information Theory, 2021, 67 (10), ⟨10.1109/TIT.2021.3094770⟩. ⟨hal-03286369⟩