Beyond stochastic gradient descent for large-scale machine learning
Staff - Faculty of Informatics
Start date: 10 March 2014
End date: 11 March 2014
The Faculty of Informatics is pleased to announce a seminar given by Francis Bach
DATE: Monday, March 10th, 2014
PLACE: USI Lugano Campus, room SI-008, Informatics building (Via G. Buffi 13)
TIME: 16.30
ABSTRACT:
Many machine learning and signal processing problems are traditionally cast as convex optimization problems. A common difficulty in solving these problems is the size of the data, where there are many observations ("large n") and each of these is large ("large p"). In this setting, online algorithms such as stochastic gradient descent which pass over the data only once, are usually preferred over batch algorithms, which require multiple passes over the data. Given n observations/iterations, the optimal convergence rates of these algorithms are O(1/\sqrt{n}) for general convex functions and reaches O(1/n) for strongly-convex functions. In this talk, I will show how the smoothness of loss functions may be used to design novel algorithms with improved behavior, both in theory and practice: in the ideal infinite-data setting, an efficient novel Newton-based stochastic approximation algorithm leads to a convergence rate of O(1/n) without strong convexity assumptions, while in the practical finite-data setting, an appropriate combination of batch and online algorithms leads to unexpected behaviors, such as a linear convergence rate for strongly convex problems, with an iteration cost similar to stochastic gradient descent. (joint work with Nicolas Le Roux, Eric Moulines and Mark Schmidt).
BIO:
Francis Bach is a researcher in the Sierra INRIA project-team, in the Computer Science Department of theEcole Normale Superieure, Paris, France. He graduated from the EcolePolytechnique, Palaiseau, France, in 1997, and earned his PhD in 2005 from the Computer Science division at the University of California, Berkeley. His research interests include machine learning, statistics, optimization, graphical models, kernel methods, sparse methods and statistical signal processing. He has been awarded a starting investigator grant from the European Research Council in 2009.
HOST: Prof. Illia Horenko
