BootBOGS: Hands-on optimizing Grid Search in hyperparameter tuning of MLP

Neural networks are widely used in the literature
in a variety of fields and for a large number of applications. A
major challenge in their use is the need to identify and process
hyperparametric values. Grid Search is a widely used technique
for meeting this task. It systematically searches for values in
a predefined range of hyperparameters. However, selecting the
appropriate range of hyperparameters can be difficult, as the
search space can be vast, resulting in an extensive number of
combinations to be tested. It is more suited to short, fast searches
for hyperparameter values, within ranges which are known to be
generally efficient. In this paper, we present an improvement to
Grid Search using a BootBOGS, a bootstrap based approach to
hyperparameter optimization. BootBOGS is a hybrid approach
that combines bootstrap and Bayesian Search with the Grid
Search technique to perform an efficient search in hyperparam eter space. Bayesian Search is used to initialize hyperparameter
ranges. Bootstrap is used to explore the distribution of model
performance for each hyperparameter combination and to reduce
its variance, enabling us to better understand the margins of
these hyperparameters and to reduce these ranges. Grid Search
is then used to refine the selection of hyperparameters. To
evaluate the effectiveness of the proposed approach, a set of
computational experiments are carried out on four different
datasets from classification problems, for which we compared
BootBOGS to several other strategies: Grid Search, Random
Search, and Bayesian Optimization. The results show that our
method is able to find better hyperparameter configurations in
terms of predictive quality with a reasonable runtime and lead
to more robust and reliable hyperparameter tuning processes