Diffusion counterfactuals with semantic abduction

In brief

• Generating counterfactual images requires answering: What would this exact image look like under a specific causal intervention?
• Standard diffusion inversion (DDIM) performs spatial abduction, but often alters patient or subject identity spuriously.
• This paper formalises diffusion counterfactuals through Pearl's causal framework, introducing spatial, semantic, and dynamic abduction to guarantee identity preservation.

This retrospective explainer is dated to the June 2025 ICML conference. It discusses Diffusion Counterfactual Generation with Semantic Abduction by Rajat Rasal, Avinash Kori, Fabio De Sousa Ribeiro, Tian Xia and Ben Glocker.

Consider a doctor evaluating a chest X-ray of an 80-year-old patient diagnosed with cardiomegaly (an enlarged heart). The doctor asks: “What would this patient’s chest scan have looked like if they were 40 years old, holding all other medical history constant?”

This is a Pearlian counterfactual query. It cannot be answered simply by pulling a random 40-year-old scan from a database, because that scan belongs to a completely different human being with different bone density, lung geometry, and thoracic proportions. The goal is to modify strictly the target causal attribute while preserving the unique, individual identity of the specific observed subject.

Diffusion models have emerged as state-of-the-art visual generators, but standard deterministic inversion (e.g. DDIM inversion) often fails this test. While it preserves high-frequency pixel noise, it lacks semantic awareness: changing an attribute like age frequently distorts facial identity, alters gender, or removes benign anatomical structures.

This paper solves this dilemma by introducing Semantic and Dynamic Abduction into diffusion architectures.

Pearl’s counterfactual recipe for images

Under Judea Pearl’s Structural Causal Model (SCM) framework, answering a counterfactual question requires three sequential mathematical steps:

1. Abduction: Infer the unobserved exogenous noise variables $\mathbf{u}$ that produced the factual observation $\mathbf{x}$ given the factual parents $\mathbf{pa}$: \(\mathbf{u} = h^{-1}(\mathbf{x}, \mathbf{pa}).\)
2. Action: Intervene on the causal graph by replacing the factual parents $\mathbf{pa}$ with the counterfactual target $\widetilde{\mathbf{pa}}$ via Pearl’s do-operator: $\mathrm{do}(\mathbf{pa} = \widetilde{\mathbf{pa}})$.
3. Prediction: Compute the counterfactual image $\widetilde{\mathbf{x}}$ by feeding the abduced noise $\mathbf{u}$ and the modified parents $\widetilde{\mathbf{pa}}$ through the structural mechanism: \(\widetilde{\mathbf{x}} = h(\mathbf{u}, \widetilde{\mathbf{pa}}).\)

In classical deep learning, autoencoders implement this by mapping pixels to a latent code $\mathbf{z}$. However, autoencoders suffer from blurry reconstructions and lack the expressive fine detail of diffusion models.

Spatial, semantic, and dynamic abduction

In diffusion models, the deterministic DDIM reverse process acts as an invertible ODE that maps an image $\mathbf{x}_0$ to a spatial noise map $\mathbf{x}_T \sim \mathcal{N}(0, I)$. Inverting this ODE constitutes Spatial Abduction (shown in panel a of the figure).

However, spatial noise $\mathbf{x}_T$ primarily encodes high-frequency textures and spatial layout; it does not capture high-level semantic identity (such as skull shape or diagnostic anatomy).

Figure from the paper. Overview of the causal mechanisms: (a) Spatial Mechanism based on DDIM inversion; (b) Semantic Mechanism with an explicit semantic encoder $e_\phi$ and latent $\mathbf{z}$; (c) Amortised Guidance combined with Dynamic Abduction ($\emptyset_{1:T}^*$) across the diffusion trajectory. Source: Rajat Rasal, Avinash Kori, Fabio De Sousa Ribeiro, Tian Xia and Ben Glocker. Extracted from the source manuscript; click to enlarge.

The authors introduce two novel mechanisms to bridge this gap:

1. Semantic Mechanism (§3.2)

An auxiliary semantic encoder $e_\phi$ extracts an identity-preserving latent vector $\mathbf{z} = e_\phi(\mathbf{x})$. During generation, the diffusion denoiser is conditioned simultaneously on the spatial noise $\mathbf{u}$, the counterfactual intervention $\widetilde{\mathbf{pa}}$, and the semantic vector $\mathbf{z}$:

\[\widetilde{\mathbf{x}} = h_\theta^\omega\left(\mathbf{u}, \widetilde{\mathbf{pa}}, \mathbf{z}\right).\]

2. Dynamic Abduction (§3.3)

During the forward inversion ODE, classifier-free guidance introduces a trajectory discrepancy between the unconditional score $\epsilon_\theta(\mathbf{x}t, \emptyset)$ and the conditional score $\epsilon\theta(\mathbf{x}t, \mathbf{pa})$. The paper amortises this discrepancy by learning a dynamic trajectory sequence $\emptyset{1:T}^*$ that dynamically steers the reverse ODE, preventing attribute leakage.

Controlled trade-offs in counterfactual generation

By parameterizing the balance between semantic guidance and spatial inversion, the framework allows clinicians and researchers to tune the trade-off between:

• Identity Preservation: Measured via cosine similarity in deep feature embeddings (e.g. ArcFace or clinical foundation encoders).
• Intervention Effectiveness: Measured by downstream classifiers verifying that the generated image satisfies the counterfactual condition $\widetilde{\mathbf{pa}}$.
• Reversibility: Verifying that applying the inverse intervention $\mathrm{do}(\widetilde{\mathbf{pa}} \to \mathbf{pa})$ recovers the exact original observation $\mathbf{x}_0$.

Experiments on both facial benchmarks (CelebA) and medical chest imaging demonstrate that semantic abduction eliminates the ghosting and identity drift common in standard diffusion inpainting, producing sharp, axiomatically sound counterfactuals.

Paper and sources

• Rajat Rasal, Avinash Kori, Fabio De Sousa Ribeiro, Tian Xia and Ben Glocker. Diffusion Counterfactual Generation with Semantic Abduction. ICML 2025 / arXiv:2506.07883.