Concept Activation Vectors

26 Oct 23 14:00 UTC

Large opaque models like neural networks require dedicated methods to study and interpret their behavior. In this series we review recent developments, analysing their relevance for business applications.

Iván has a PhD in theoretical physics from Florence University. After spending a few years doing research in different topics in quantum physics, he became interested in AI at the Max Planck Institute of Quantum Optics, where he applied Deep Neural Network techniques to study different properties of complex quantum materials. After his journey in academia he decided to move to the private sector in order to apply his experience in AI, physics and mathematics to challenging real life problems. Before joining appliedAI in 2021, he worked at a startup applying AI and GPGPU computing to optimize CNC-manufacturing processes.

In the last seminar of our XAI series, Iván Rodríguez from appliedAI talks about Concept Activation Vectors (CAVs). CAVs go beyond feature attribution and bring a quantitative approach to testing. He will discuss how this tool interprets a neural network’s internal state in terms of human-friendly concepts.

Abstract: In this XAI seminar, we’ll start by diving into the testing with Concept Activation Vectors (TCAV) method, which helps us gauge how much a model’s prediction is influenced by a user-defined concept. Additionally, we’ll explore two other papers, Concept Bottleneck Models (CBM) and Post-Hoc CBM, which present clever approaches to engage with user-defined concepts. For instance, if the model initially overlooked a bone spur on an X-ray, would it still predict severe arthritis?

References

[Kim18I]

Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV), Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, Rory Sayres.

Jun 2018

The interpretation of deep learning models is a challenge due to their size, complexity, and often opaque internal state. In addition, many systems, such as image classifiers, operate on low-level features rather than high-level concepts. To address these challenges, we introduce Concept Activation Vectors (CAVs), which provide an interpretation of a neural net's internal state in terms of …

[Koh20C]

Concept Bottleneck Models, Pang Wei Koh, Thao Nguyen, Yew Siang Tang, Stephen Mussmann, Emma Pierson, Been Kim, Percy Liang.

Nov 2020

We seek to learn models that we can interact with using high-level concepts: if the model did not think there was a bone spur in the x-ray, would it still predict severe arthritis? State-of-the-art models today do not typically support the manipulation of concepts like "the existence of bone spurs", as they are trained end-to-end to go directly from raw input (e.g., pixels) to output (e.g., …

[Yuk23P]

Post-hoc Concept Bottleneck Models, Mert Yuksekgonul, Maggie Wang, James Zou.

Feb 2023

Concept Bottleneck Models (CBMs) map the inputs onto a set of interpretable concepts (``the bottleneck'') and use the concepts to make predictions. A concept bottleneck enhances interpretability since it can be investigated to understand what concepts the model "sees" in an input and which of these concepts are deemed important. However, CBMs are restrictive in practice as they require dense …

References

In this series →