Slowness Learning for Curiosity-Driven Agents

You are cordially invited to attend the PhD Dissertation Defense of Varun KOMPELLA on Thursday, December 18th 2014 at 15h30in room 003 (Informatics building)

Abstract:

In the absence of external guidance, how can a robot learn  to map the many raw pixels of high-dimensional visual inputs to useful action sequences? I study methods that  achieve this by making robots self-motivated (curious) to  continually build compact representations of sensory  inputs that encode different aspects of the changing environment.

Previous curiosity-based agents acquired skills by  associating intrinsic rewards with world model  improvements, and used reinforcement learning (RL) to  learn how to get these intrinsic rewards.  But unlike in  previous implementations, I consider streams of  high-dimensional visual inputs, where the world model is a  set of compact low-dimensional representations of the  high-dimensional inputs. To learn these representations, I  use the \emph{slowness learning} principle, which states  that the underlying causes of the changing sensory inputs  vary on a much slower time scale than the observed sensory  inputs. The representations learned through the slowness  learning principle are called slow features (SFs). Slow features have been shown to be useful for RL, since they  capture the underlying transition process by extracting  spatio-temporal regularities in the raw sensory inputs.

However, existing techniques that learn slow features are  not readily applicable to curiosity-driven online learning  agents, as they estimate computationally expensive  covariance matrices from the data via batch processing.

The first contribution called the incremental SFA  (IncSFA), is a low-complex, online algorithm that extracts  slow features without storing any input data or estimating  costly covariance matrices, thereby making it suitable to  be used for several online learning applications.

However, IncSFA gradually forgets previously learned  representations whenever the statistics of the input  change. In open-ended online learning, it becomes  essential to store learned representations to avoid  re-learning previously learned inputs.

The second contribution is an online active modular IncSFA  algorithm called the curiosity-driven modular incremental  slow feature  analysis (Curious Dr.\ MISFA). Curious Dr.\  MISFA  addresses the forgetting problem faced by IncSFA and learns expert slow feature abstractions in order from  least to most costly.

The third contribution uses the Curious Dr.\ MISFA algorithm in a continual curiosity-driven skill acquisition framework that enables robots to acquire, store, and re-use both abstractions and skills in an online and continual manner.

I provide (a) a formal analysis to the working of the  proposed algorithms; (b) compare them to the existing  methods; and (c) use the iCub humanoid robot to  demonstrate their application in real-world environments.

These contributions together demonstrate that the online implementations of slowness learning make it suitable for  an open-ended curiosity-driven RL agent to acquire a  repertoire of skills that map the many raw pixels of  high-dimensional images to multiple sets of action  sequences.

Dissertation Committee:

- Prof. Juergen Schmidhuber, Università della Svizzera italiana, Switzerland (Research Advisor)

- Prof. Stefan Wolf, Università della Svizzera italiana, Switzerland (Internal Member)

- Prof. Matthias Hauswirth, Università della Svizzera italiana, Switzerland (Internal Member)

- Prof. Laurenz Wiskott, Ruhr-Universität Bochum, Germany (External Member)

- Prof. Srini Narayanan, Google-Zurich & UC-Berkeley, USA (External Member)

- Prof. Benjamin Kuipers, University of Michigan, USA (External Member)