Causal composition in object-centric representations

In brief

• Object-Centric Learning (OCL) decomposes scenes into object slots, but often fails to model causal dependencies between them.
• Scene graph methods structure relationships but typically rely on dense human annotations.
• This paper establishes a probabilistic bridge between causality and object-centric learning via causal abstraction, ensuring that scene compositions obey invariant physical laws under intervention.

This retrospective explainer is dated to the April 2025 ICLR World Models Workshop. It discusses Unifying Causal and Object-centric Representation Learning allows Causal Composition by Avinash Kori, Ben Glocker, Bernhard Schölkopf and Francesco Locatello.

Imagine rearranging furniture in a living room. If you pick up a floor lamp and move it from the corner to the center of the room, you expect the light pool and shadows cast across the carpet to move with it. You would be bewildered if moving the lamp spontaneously changed the color of the sofa or altered the shape of the coffee table.

Human cognition takes this modularity for granted: the world is composed of discrete, bounded objects whose internal properties remain stable even when their spatial and relational arrangements are altered.

In deep generative modeling, however, this property—known as causal compositionality—is remarkably difficult to achieve. Standard Object-Centric Learning (OCL) models decompose an image into latent slot vectors, but they treat slots either as completely independent entities or as unconstrained continuous vectors. When a model tries to compose a novel scene by editing one object, it frequently hallucinates unintended side effects across unrelated objects.

This paper bridges Object-Centric Learning with Causal Representation Learning (CRL), formalizing how object slots must be structured to support valid causal interventions.

Why object slots alone are insufficient

Standard Slot Attention binds visual tokens into slots $s_1, \dots, s_K$. In theory, slot $s_k$ represents object $k$. However:

1. Semantic confounding: In a real scene, objects interact. A red cube sitting on top of a yellow sphere casts a shadow and reflects red light onto the yellow surface. If an autoencoder treats slot representations as statistically independent, it cannot model these physical interactions.
2. Global entanglement: Conversely, if the decoder uses global attention across all slots, an intervention on slot $s_1$ (e.g. changing the cube’s position) leaks gradients that distort the appearance of slot $s_2$.
3. Absence of structural mechanisms: Standard OCL lacks an explicit graph structure dictating which objects causally influence which others.

Grounding scene graphs in causal abstraction

To resolve this, the paper formulates Scene Graph Modeling (SGM) through the lens of Structural Causal Models (SCMs).

A scene consists of $K$ objects $O_1, \dots, O_K$. Each object entity possesses internal intrinsic attributes $A_i$ (such as color, material, and geometric shape) and extrinsic relational attributes $R_i$ (such as 3D coordinate location, support relations, and occlusion order).

The scene generative process is governed by a causal graph $\mathcal{G} = (\mathcal{V}, \mathcal{E})$ where each node is an object entity:

\[O_i = f_i\left(\mathbf{pa}_{\mathcal{G}}(O_i), U_i\right),\]

where $\mathbf{pa}_{\mathcal{G}}(O_i)$ are the causal parents of object $O_i$ in the scene (for example, the table supporting the cup), $U_i$ represents exogenous independent noise, and $f_i$ is an autonomous, invariant causal mechanism.

The foundational principle of causality is the Independent Causal Mechanisms (ICM) assumption: the mechanism $f_i$ that determines object $O_i$ given its parents operates independently of all other mechanisms $f_j$ ($j \neq i$).

What causal composition enables

When a user or an autonomous agent intervenes on an object via Pearl’s do-operator:

\[\mathrm{do}\left(O_k = o^*\right),\]

the modularity of the SCM guarantees that only the target object $O_k$ and its causal descendants in $\mathcal{G}$ are updated:

\[p\left(O_j \mid \mathbf{pa}(O_j); \mathrm{do}(O_k = o^*)\right) = p\left(O_j \mid \mathbf{pa}(O_j)\right) \quad \forall j \notin \mathrm{desc}(O_k).\]

All non-descendant objects remain strictly invariant.

This theoretical unification provides concrete design rules for building next-generation visual world models:

• Disentangled Slot Encoders: Slot initialisations must capture intrinsic object properties without baking in extrinsic environmental lighting.
• Explicit Relational Edges: Object interactions must be routed through explicit directed mechanisms rather than unconstrained cross-attention.
• Counterfactual Soundness: An agent predicting the consequences of an action (such as pushing a block) can simulate downstream collisions without corrupting static background objects.

Paper and sources

• Avinash Kori, Ben Glocker, Bernhard Schölkopf and Francesco Locatello. Unifying Causal and Object-centric Representation Learning allows Causal Composition. ICLR 2025 Workshop on World Models.