DIET: Machine Unlearning on a Data-Diet

“In the practical use of our intellect, forgetting is as important a function as recollecting.”

Why forgetting without retain data is hard

Machine unlearning asks a trained model to remove a targeted concept without damaging everything else it has learned. Most methods protect that surrounding knowledge by training against a retain set. In practice, the original data may be private, unavailable, or too expensive to reconstruct.

The problem is sharper in vision-language models: a concept is encoded through the alignment between images and language. Breaking that alignment too aggressively can distort nearby concepts; changing it too little leaves the target knowledge intact.

A geometric route to selective forgetting

DIET maps image and text embeddings into a Poincaré ball. Text prompts for classes that should remain provide semantic boundary prototypes. Adaptive optimal transport assigns each forget example a suitable mismatched prototype, defining where its embedding should move rather than merely pushing it in an arbitrary direction.

Because the boundary lies at infinite geodesic distance, DIET uses the Busemann function as a finite, differentiable objective:

The final objective balances three behaviours:

Only low-rank adapters in the vision encoder are updated. The text encoder supplies the prototypes, so the method does not need a retained image dataset as a training constraint.

What the geometry changes

The latent-space analysis makes the trade-off visible. GS-LoRA and its retain-free variant drive target similarity to negative values, while the retain-free variant also disperses the surrounding clusters. DIET instead follows a structured boundary direction and preserves the organisation of retained concepts.

Evaluation at a glance

The study evaluates CLIP on Flowers102, OxfordPets, StanfordCars, and Food101. Each run uses 16 forget samples per concept and averages results over five randomly selected forget concepts per dataset. Lower forget-set accuracy (D_f) is better; higher test accuracy (D_t) means more general utility is retained.

Average accuracy across the four fine-grained datasets in Table 1. An asterisk marks a baseline adapted to run without retained data.

DIET retains 25.51 percentage points more test accuracy than retain-free SalUn and comes within 2.72 points of retain-based GS-LoRA. The comparison also exposes why forget accuracy cannot be read alone: GS-LoRA* reaches zero forget accuracy but collapses overall test accuracy to 1.73%.

What matters in the method

The ablations support two design choices. Adaptive optimal transport gives each forget example a meaningful boundary direction, while the repulsive loss protects unassigned semantic prototypes. Replacing text-derived prototypes with random boundary points weakens forgetting across every reported dataset, even when retained-class separation remains high.

Performance also depends on the number of forget examples. DIET preserves useful accuracy with very small sample sets, but its forgetting becomes less reliable below the 16-shot setting. Determining how much evidence is needed to remove a concept remains open.

Scope and limitations

• DIET attenuates image–text alignment; it does not prove complete removal of a concept from every internal representation or downstream behaviour.
• Hyperbolic optimisation can be unstable and requires norm clipping, a repulsive regulariser, and careful hyperparameter tuning.
• Results depend on the quality of the text prototypes, and the boundary construction may be insufficient for heavily entangled concepts.
• The experiments cover targeted concept removal in CLIP-style classification settings. Applications with legal or ethical requirements for verified deletion need stronger auditing and guarantees.

Takeaway

DIET turns retain-data-free unlearning into a geometric routing problem: choose a semantically meaningful direction for each target example, move it toward the hyperbolic boundary, and protect the concepts that should remain. The result is selective forgetting that preserves substantially more utility than the retain-free baselines tested in the paper.