TRuST: selective fine-tuning for robust concept unlearning in diffusion models

In brief

• Text-to-image diffusion models can be manipulated by malicious prompts to generate harmful, copyrighted, or sexually explicit content.
• Existing unlearning methods either fine-tune all parameters (degrading benign image quality) or compute a static parameter mask once (easily defeated by adaptive adversarial attacks).
• TRuST (Targeted Robust Selective fine Tuning) dynamically tracks concept neurons at every optimization step, combining sensitivity reduction with Hessian regularization.

This retrospective explainer is dated to the February 2026 preprint release, subsequently appearing at ACM CCS 2026. It discusses Selective Fine-Tuning for Targeted and Robust Concept Unlearning by Mansi, Avinash Kori, Francesca Toni and Soteris Demetriou.

Consider a state-of-the-art text-to-image diffusion model trained on billions of internet images. The model can render photorealistic landscapes, architectural schematics, and family portraits. Unfortunately, it also memorizes copyrighted artist styles, explicit adult content, and instructions for fabricating dangerous chemicals.

If a developer wants to “unlearn” a harmful concept (say, generating violent imagery), the naive approach is to fine-tune the entire network on negative objectives. But fine-tuning hundreds of millions of weights acts like blunt chemotherapy: it destroys the model’s ability to render benign everyday concepts (e.g., rendering a ham sandwich or a wooden table starts producing distorted mush).

To minimize collateral damage, researchers developed parameter-efficient unlearning methods that identify a subset of weights to edit. However, existing methods compute a static saliency mask once. While this stops simple static prompts, an adaptive adversary using gradient-guided prompt optimization easily circumvents the static barrier.

This paper introduces TRuST (Targeted Robust Selective fine Tuning), a framework that dynamically tracks concept-specific neurons at every optimization step while protecting benign model capabilities through Hessian-based regularization.

Why static unlearning masks fail against adaptive attacks

The paper analyzes why popular methods like Saliency-based Unlearning (SalUn) fail under adversarial evaluation:

Figure from the paper. (a) Static vs adaptive adversaries attacking safeguarded diffusion models; (b) Comparison of SalUn's static mask (computed once) vs TRuST's dynamic mask (updated per step), alongside Attack Success Rate (ASR ↓) under adaptive adversaries (14.6% for SalUn vs 0.27% for TRuST); (c) The TRuST pipeline combining Concept Influence Penalty Loss, Concept Sensitivity Reduction Loss, and Hessian regularization. Source: Mansi, Avinash Kori, Francesca Toni and Soteris Demetriou. Extracted from the source manuscript; click to enlarge.

1. Static Masking (SalUn): SalUn computes a binary parameter mask $M$ once at the start of training based on initial gradients. As fine-tuning proceeds, the model’s internal representation shifts, but the mask remains locked in place. An adaptive adversary easily finds “jailbreak” prompt vectors that bypass the locked neurons. Under adaptive attacks, SalUn’s Attack Success Rate (ASR) jumps to 14.6%.
2. Dynamic Masking (TRuST): TRuST re-evaluates the gradient sensitivity of each cross-attention layer at every single fine-tuning step. As the model adapts, the mask dynamically tracks where the harmful concept attempts to relocate, crushing the adaptive ASR down to 0.27%.

The TRuST optimization objective

To simultaneously eliminate the concept and preserve general visual fidelity, TRuST optimizes three complementary loss functions:

1. Concept Influence Penalty Loss ($\mathcal{L}_{\text{CIP}}$)

Pushes the model’s predicted noise under the harmful concept $c_{\text{harm}}$ toward a neutral, benign reference target $c_{\text{benign}}$:

\[\mathcal{L}_{\text{CIP}} = \mathbb{E}_{x_t, t} \left[ \left\| \epsilon_\theta(x_t, t, c_{\text{harm}}) - \epsilon_\theta(x_t, t, c_{\text{benign}}) \right\|_2^2 \right].\]

2. Concept Sensitivity Reduction Loss ($\mathcal{L}_{\text{CSR}}$)

Flattens the local loss surface around the harmful prompt embedding, ensuring that slight prompt perturbations (e.g. typos, homophones, or adversarial embedding noise) cannot re-activate the concept:

\[\mathcal{L}_{\text{CSR}} = \mathbb{E}_{x_t, t} \left[ \left\| \nabla_c \, \epsilon_\theta(x_t, t, c_{\text{harm}}) \right\|_2^2 \right].\]

3. Hessian-based Benign Regularization

To guarantee that benign knowledge is not eroded, TRuST applies a quadratic penalty weighted by the empirical Fisher Information Matrix (approximating the local Hessian $H$):

\[\mathcal{R}_{\text{benign}}(\Delta \theta) = \frac{1}{2} \, \Delta \theta_M^T \, H_{\text{benign}} \, \Delta \theta_M,\]

where $\Delta \theta_M = M \odot (\theta - \theta_0)$ represents the weight updates strictly within the dynamic mask $M$. Weights that are critical for general image generation are heavily penalized if modified.

Beyond single concepts: combinations and conditionals

Real-world safety violations rarely involve isolated keywords. Malicious users construct concept combinations (e.g., combining two benign objects in a hazardous configuration) or conditional concepts (e.g., medical anatomy that is permissible in an educational context but prohibited in an offensive context).

The authors demonstrate that because TRuST targets the underlying cross-attention concept neurons dynamically, it achieves robust unlearning of:

• Multi-concept combinations without requiring combinatorial training pairs.
• Conditional concepts with clean boundary preservation.
• Over 50% faster training convergence than full fine-tuning baselines, with negligible degradation in Frechet Inception Distance (FID) on standard COCO benchmarks.

Paper and sources

• Mansi, Avinash Kori, Francesca Toni and Soteris Demetriou. Selective Fine-Tuning for Targeted and Robust Concept Unlearning. ACM CCS 2026 / arXiv:2602.07919.