Learning hidden variables in a causal order
In brief
- The paper studies representations whose hidden variables can influence one another.
- A causal ordering prior constrains the latent representation under additive-noise assumptions.
- Identifiability depends on the model assumptions; reconstructing images alone is insufficient.
This retrospective explainer is dated to the July 2023 preprint release. It discusses A Causal Ordering Prior for Unsupervised Representation Learning by Avinash Kori, Pedro Sanchez, Konstantinos Vilouras, Ben Glocker and Sotirios A. Tsaftaris.
Consider a photograph of a pendulum and its shadow. The pendulum’s position and the light source affect the shadow. Treating all three as unrelated image attributes loses something important: changing a cause can change its effects.
A representation learner compresses an observation, such as an image, into hidden variables. Those variables are called latent because the model learns them rather than receiving their values directly. The question in this paper is whether the latent variables can be organised to reflect causal relationships instead of merely providing a convenient compression.
The proposed approach uses a causal ordering prior. It studies what becomes identifiable when the latent data-generating process has an appropriate structure and develops a learning objective intended to encourage that order.
Why independence is not always the right target
Many representation-learning approaches encourage independent latent dimensions. Independence is attractive because it discourages different coordinates from redundantly encoding the same information. But causes and effects are generally dependent.
In the pendulum example, a representation that stores position and shadow location should preserve their relationship. Forcing an independent representation might instead encode the independent sources of variation, or find another transformation that is useful for reconstruction but harder to interpret as the variables of interest.
This does not make independence a universally bad assumption. It means the assumption must match the desired representation. Recovering independent noise sources and recovering causally related variables are different tasks.
The figure illustrates the paper’s motivation: latent structure should be able to represent dependence, rather than assuming that every meaningful factor varies independently of the others.
The assumed data-generating process
The paper describes observations using:
\[x=f_o(z)+\epsilon_x, \qquad P(z)=\prod_i P(z_i\mid\mathrm{pa}(z_i)).\]Here, x is an observation, z is its latent representation, and the function f_o maps latent variables into the observed space. The term epsilon_x represents observation noise. The parents of z_i, written pa(z_i), are its direct causes in a directed acyclic graph.
The second expression factorises the latent distribution according to that graph. It allows one hidden variable to depend on others. A directed acyclic graph has no directed loop: following causal arrows cannot bring us back to the same variable.
The paper further assumes additive-noise mechanisms:
\[z_i:=f_i(\mathrm{pa}(z_i))+\epsilon_i.\]Each variable is generated from a function of its parents plus exogenous noise. “Exogenous” means that the noise enters from outside the modelled causal relationships. This is a substantive restriction on how the hidden variables are generated, not a general description of every possible system.
What a causal order tells us
A causal order places a variable before its descendants. Imagine a recipe in which mixing ingredients must precede baking, and baking must precede serving. The order constrains which steps could influence later ones, but does not specify every dependency or quantity.
Likewise, knowing a causal order is weaker than knowing the complete graph. Several graphs can respect the same order. Some variables may be unrelated, and multiple valid orders may exist.
The paper separates these cases in its theory. Stronger structural information can remove more ambiguity in the representation. An ordering constraint permits weaker equivalences than complete recovery of every coordinate with a uniquely fixed meaning.
That qualification matters because identifiability is always identifiability of something, under specified assumptions and up to specified transformations. It is not a guarantee that a neural network assigns the same numerical vector in every training run.
How the ordering constraint is learned
The method models the latent distribution with a Gaussian mixture and uses a variational framework to learn the representation. A Gaussian mixture combines several component distributions, giving a more flexible latent density than one Gaussian.
The ordering objective uses properties of additive-noise models. Under the paper’s additional noise assumptions, the behaviour of second derivatives of the latent log-density helps distinguish leaf variables: variables with no children in the causal graph.
The resulting loss encourages an organisation in which successive latent coordinates satisfy the appropriate ordering property. This is different from receiving the correct graph as a label. The training procedure attempts to impose a property that the assumed causal process should possess.
A useful analogy is arranging a set of tasks by repeatedly finding tasks that have no remaining dependants. The analogy explains the ordering idea, but the implementation works with estimated properties of a probability density, not a known list of dependencies.
The paper uses Stein kernel estimators to approximate the required derivatives. Estimation quality therefore matters. A theoretical property of an exact distribution need not be recovered perfectly from finite data and a learned latent space.
What the experiments measure
The experiments include synthetic tabular data and image datasets such as MorphoMNIST and Causal3DIdent. The method is compared with representation-learning baselines, including beta-VAE and a mixture-based variational autoencoder.
The evaluation asks more than whether an image can be reconstructed. Mean correlation coefficient measures agreement between learned variables and reference factors, or stability between runs after appropriate alignment. Causal ordering divergence examines whether the learned arrangement matches the reference ordering.
Both are needed. A representation can contain useful information without arranging it in the proposed order. It can also appear ordered while representing the wrong factors. A reconstruction score alone would not expose either distinction.
Synthetic data are especially useful here because the generating variables and graph are known. That makes it possible to test recovery directly. Results on such data remain conditional on the relationship between the simulator and the assumptions used by the method.
Where the claim stops
The theory assumes a restricted observation mapping and latent causal process. The paper also notes that its formulation treats the latent features as direct parents of the observation, so it cannot recover every indirect cause outside that representation.
In practice, unmeasured variables, unsuitable noise distributions or an inaccurate decoder can change what the method recovers. A low ordering loss is therefore not a certificate that the learned variables are the true causes in an arbitrary real-world dataset.
The contribution is a concrete way to connect representation learning with causal ordering, together with an account of the assumptions that make this connection meaningful. It gives a more precise question than “are these features disentangled?”: do they recover the intended variables, preserve their dependence and organise them in an order compatible with the generating process?
Paper and sources
- Avinash Kori, Pedro Sanchez, Konstantinos Vilouras, Ben Glocker and Sotirios A. Tsaftaris. A Causal Ordering Prior for Unsupervised Representation Learning. arXiv:2307.05704 (2023).