Robustly representing uncertainty through sampling in deep neural networks

05 Dec 22 00:00 UTC

Uncertainty quantification (UQ) in machine learning is the practice of measuring or estimating uncertainty in models. It is a set of tools to understand the limitations of models and predictions, and to make better decisions. UQ is a key …

Different types to sample units and connections in neural networks. Each strategy can be used for uncertainty quantification.

Fabio Peruzzo (appliedAI Initiative)

After receiving his PhD in Applied Mathematics with a specialization in complex systems, Fabio was drawn to the potential of the burgeoning field of machine learning. This interest led him to transition into AI engineering. His professional journey includes roles at several startups in the UK. As of 2021, he has been contributing his expertise to the appliedAI Transferlab team. His primary areas of research encompass data valuation, interpretable AI, and reinforcement learning.

In the previous instalments of our seminar series we have explored how bayesian methods can be used to estimate the epistemic uncertainty of a model, i.e. the uncertainty coming from the choice of network’s parameters. In particular, two weeks ago we have seen how variational inference applied to Bernoulli and Gaussian dropout can make Bayesian NNs tractable under certain assumptions. This week’s seminar will focus on a few numerical experiments that highlight benefits and limitations of this approach. We will start by giving a brief summary of the main mathematical results that we have seen so far, then discuss the difference between dropout and dropconnect and end with an analysis of their robustness to noise when applied to CNNs trained on the MNIST and Cifar-10. Throughout the talk we will use the paper “Robustly representing uncertainty through sampling in deep neural networks” as our main reference.

References

[Mcc18R]

Robustly representing uncertainty in deep neural networks through sampling, Patrick McClure, Nikolaus Kriegeskorte.

Jan 2018

As deep neural networks (DNNs) are applied to increasingly challenging problems, they will need to be able to represent their own uncertainty. Modeling uncertainty is one of the key features of Bayesian methods. Using Bernoulli dropout with sampling at prediction time has recently been proposed as an efficient and well performing variational inference method for DNNs. However, sampling from other …

Publication

[Gal16D]

Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning, Yarin Gal, Zoubin Ghahramani.

Oct 2016

Deep learning tools have gained tremendous attention in applied machine learning. However such tools for regression and classification do not capture model uncertainty. In comparison, Bayesian models offer a mathematically grounded framework to reason about model uncertainty, but usually come with a prohibitive computational cost. In this paper we develop a new theoretical framework casting …

[Mob21D]

DropConnect is effective in modeling uncertainty of Bayesian deep networks, Aryan Mobiny, Pengyu Yuan, Supratik K. Moulik, Naveen Garg, Carol C. Wu, Hien Van Nguyen.

Mar 2021

Deep neural networks (DNNs) have achieved state-of-the-art performance in many important domains, including medical diagnosis, security, and autonomous driving. In domains where safety is highly critical, an erroneous decision can result in serious consequences. While a perfect prediction accuracy is not always achievable, recent work on Bayesian deep networks shows that it is possible to know …

Publication

References

In this series →