Are They the Same Picture? Adapting Concept Bottleneck Models for Human-AI Collaboration in Image Retrieval

IJCAI 2024 [Top 5% in Human-Centered AI Track]
1University of Michigan
, 2Massachusetts Institute of Technology

Abstract

Image retrieval plays a pivotal role in wildlife conservation, specifically in state-of-the-art platforms like ElephantBook for finding individual animals. Although deep learning techniques for image retrieval have advanced significantly, their imperfect real-world performance often necessitates including human expertise. Human-in-the-loop approaches typically rely on humans completing the task independently and then combining their opinions with an AI model in various ways, as these models offer very little interpretability or correctability. To allow humans to intervene in the AI model instead, thereby saving human time and effort, we adapt the Concept Bottleneck Model (CBM) and propose CHAIR. CHAIR (a) enables humans to correct intermediate concepts, which helps improve embeddings generated, and (b) allows for flexible levels of intervention that accommodate varying levels of human expertise for better retrieval. To show the efficacy of CHAIR, we demonstrate that our method performs better than similar models on image retrieval metrics without any external intervention. Furthermore, we also showcase how human intervention helps further improve retrieval performance, thereby achieving human-AI complementarity.

CHAIR enables human-AI collaborative image retrieval pipelines

CHAIR Architecture

CHAIR Architecture

CHAIR Architecture - Fuses embeddings with concept projection onto the embedding space

CHAIR vs Baselines in Image Retrieval

Improvement of CHAIR over baseline

CHAIR improvement with human intervention in Image Retrieval

Improvement of CHAIR over baseline

TSNE plot of the embeddings with increasing human intervention

TSNE Plot

BibTeX

@inproceedings{balloli2024chair,
        title     = {Are They the Same Picture? Adapting Concept Bottleneck Models for Human-AI Collaboration in Image Retrieval},
        author    = {Balloli, Vaibhav and Beery, Sara and Bondi-Kelly, Elizabeth},
        booktitle = {Proceedings of the Thirty-Second International Joint Conference on
                     Artificial Intelligence, {IJCAI-24}},
        publisher = {International Joint Conferences on Artificial Intelligence Organization},
        year      = {2024},
        note     = {Human-Centered Artificial Intelligence},
      }