Extrapolating hyperparameters across dataset size with Bayesian optimisation

11 Apr 18 00:00 UTC

Álvaro Tejero-Cantero (University of Tübingen)

Álvaro’s career journey spans across the fields of theoretical physics, computational neuroscience, and machine learning. Initially focusing on the interference of entangled material particles, and later on models of memory sequences through theoretical neural-network models and data analysis, his work has been deeply interdisciplinary. His experience includes a stint at the German Aerospace Agency and the European Southern Observatory. During a postdoc in Oxford, he developed an interest in machine learning, applying Bayesian time-series models to classical signal analysis techniques. Álvaro’s professional path has oscillated between academia and industry, exploring the balance between theory and practical application across various scientific fields. His recent work involves applying machine learning to mechanistic models for solving inverse problems through simulation-based inference methods.

At the ml ⇌ science colab, Álvaro and his team leverage machine learning methods for scientific discovery in fields as diverse as archaeology, paleo-climatology and high-energy physics.

In this talk, Álvaro introduces us to FABOLAS, a Bayesian optimization procedure for hyperparameter tuning “which models loss and training time as a function of dataset size and automatically trades off high information gain about the global optimum against computational cost.” This is done with “a generative model for the validation error as a function of training set size, which is learned during the optimization process and allows exploration of preliminary configurations on small subsets, by extrapolating to the full dataset.”

References

[Kle17F]

Fast Bayesian Optimization of Machine Learning Hyperparameters on Large Datasets, Aaron Klein, Stefan Falkner, Simon Bartels, Philipp Hennig, Frank Hutter.

Apr 2017

Bayesian optimization has become a successful tool for hyperparameter optimization of machine learning algorithms, such as support vector machines or deep neural networks. Despite its success, for ...

Publication