State representations hold the promise of simplifying control, allowing a reinforcement learning (RL) agent to solve a task more quickly, and to generalize better to new tasks. While this representation can be learned in a multi-task setting, doing so requires manually constructing a suitable task distribution, an onerous requirement. Instead, we propose to learn a representation that encodes as few bits of the input as possible, subject to the constraint that the agent is still able to solve this task. This essentially amounts to placing “blinkers” on our agent, with the aim of ignoring spurious attributes of the state. Formally, we adopt the information bottleneck (IB) as a measure of representational complexity, and augment the standard RL objective with a lower bound.
Emergence of Invariance and Disentanglement in Deep Representations,
Using established principles from Statistics and Information Theory, we show that invariance to nuisance factors in a deep neural network is equivalent to information minimality of the learned representation, and that stacking layers and injecting noise during training naturally bias the network towards learning invariant representations. We then decompose the cross-entropy loss used during …