Grounded Object-Centric Learning: giving slots a reusable vocabulary
In brief
- Conditional Slot Attention learns a Grounded Slot Dictionary for reusable object representations.
- The dictionary connects object types with distributions used to initialise object slots.
- The paper tests scene composition and task adaptation as well as object discovery.
This retrospective explainer is dated to the May 2024 ICLR conference; the linked manuscript began in July 2023. It discusses Grounded Object-Centric Learning by Avinash Kori, Francesco Locatello, Fabio De Sousa Ribeiro, Francesca Toni and Ben Glocker.
Suppose you sort the objects on a desk into separate trays. Separating the pen, mug and notebook is useful. But if the trays have a different meaning every time you look at the desk, it is harder to reuse what you have learned about each object.
Object-centric learning faces a similar problem. A model can divide an image into object representations without learning a stable vocabulary that connects those representations across scenes. Grounded Object-Centric Learning addresses this distinction through COnditional Slot Attention, abbreviated COSA in the paper, and a Grounded Slot Dictionary.
An earlier version appeared in 2023 as “Unsupervised conditional slot attention for object centric learning.” This post covers the ICLR 2024 paper and that earlier Scholar entry together, rather than presenting them as separate methods.
What is a slot?
A slot is a vector intended to represent one object or component of a scene. Slot Attention starts with a set of slots and repeatedly updates them by attending to image features. Features compete for assignment, allowing different slots to specialise in different parts of an image.
A decoder then uses those representations to reconstruct the scene. If the reconstruction works and the slots correspond to objects, the model has found a useful decomposition without being given an object mask for every training image.
Standard Slot Attention samples its initial slots from one shared distribution. This treats the slots symmetrically, which is useful for unordered sets of objects. It does not, however, explicitly encourage a reusable association between a particular object type and a particular initialisation distribution.
The desk analogy therefore needs a refinement. The problem is not that a mug must always occupy tray number one. Slot indices can be permuted. The problem is whether the model can recognise and reuse a representation of mug-like content across different scenes.
A Grounded Slot Dictionary
The paper introduces a dictionary containing canonical object-level property vectors and associated Gaussian distributions. These distributions supply initial conditions for slots before iterative refinement.
“Canonical” refers to a representation intended to capture recurring object properties rather than every detail of one appearance. “Grounded” means that the dictionary is learned from the visual data. The dictionary is not a list of human-written labels such as mug, pen and notebook.
A library catalogue offers a useful analogy. A catalogue entry describes a recurring kind of item, while an individual copy has its own location and condition. Similarly, the dictionary provides a reusable starting point, and refinement adapts that starting point to the particular instance in the current image.
The figure shows why dictionary learning is only part of the system. The model must first extract useful features, select appropriate initialisations, bind them to the current scene and decode or use the resulting slots.
How the pieces fit together
The paper divides the architecture into five modules. The abstraction module uses spectral decomposition to identify distinct content in the encoded scene. The Grounded Slot Dictionary maps this information to canonical representations and slot distributions.
The refinement module then applies Slot Attention. The discovery module reconstructs observations from the resulting slots. For downstream tasks, a reasoning module transforms object properties and predicts the required output.
The ordering of these operations matters. Dictionary entries do not replace attention. They condition its starting point so that iterative binding can begin with more structured information than samples from one shared distribution.
The model also considers the number of unique and repeated objects. Repeated objects make this task subtle: a scene can contain two objects of the same type. A vocabulary entry is therefore not the same thing as a unique instance. Counting types and counting objects must remain distinguishable.
Why sampling from a distribution helps
An object type has variation. Two mugs can differ in shape, orientation and appearance while still sharing useful properties. Assigning a distribution to a dictionary entry allows the initial slot to vary rather than forcing every instance onto an identical vector.
The refinement process then incorporates evidence from the current image. This balances a reusable representation with instance-specific information. A dictionary that ignored the current scene would be too rigid; a representation with no reusable structure would lose the motivation for grounding.
The paper gives a probabilistic account of this construction and derives an end-to-end objective using variational inference. The important conceptual point is that the dictionary and reconstruction process are trained together. The dictionary is expected to support the image model, rather than being attached afterwards merely to name clusters.
No additional equation is needed to understand this distinction. The architectural diagram makes the conditioning relationship more directly visible than a long reconstruction of the training objective would.
What can grounded slots be used for?
Object discovery asks whether the slots separate the image into useful components. Scene generation asks whether representations can produce plausible scenes. Scene composition asks whether selected object representations can be combined into new arrangements.
Task adaptation asks whether the learned object structure remains useful when the downstream requirement changes. A model that only reconstructs its training images well might still struggle with these other tasks.
The paper therefore evaluates several uses of the representation. Its claim is not simply that a dictionary improves every segmentation metric. The experiments report benefits for generation, composition and task adaptation while remaining competitive with standard Slot Attention on object discovery benchmarks.
Consider composing a scene containing two objects whose types appeared separately during training. Reusable representations provide a possible mechanism for recombining them. But a plausible composition is still an empirical result to test; the mere existence of dictionary entries does not guarantee that every new combination will be valid.
What “grounded” does not guarantee
A learned dictionary may be consistent without matching a human taxonomy. It may group objects by a property useful for reconstruction rather than by the category a reader would choose. Visual inspection and downstream evaluation help determine what an entry actually represents.
Similarly, compositional generation is not automatically causal reasoning. Rearranging representations can produce a new image without specifying how an intervention should affect other objects or their relationships.
There are also ordinary modelling choices: dictionary capacity, feature quality and the behaviour of the decoder affect what can be learned. A finite vocabulary may be too coarse for some distinctions or unnecessarily fragmented for others.
The contribution is to make a particular part of object-centric learning explicit: binding an object instance should be connected to a reusable representation of its type. COSA provides an architecture for learning that connection and evaluates whether it helps beyond the immediate task of separating an image into slots.
Paper and sources
- Avinash Kori, Francesco Locatello, Fabio De Sousa Ribeiro, Francesca Toni and Ben Glocker. Grounded Object-Centric Learning. ICLR 2024.