Evaluating LLM Unlearning via Comparison of Internal Representations

One-Line Summary

A study on how to tell whether knowledge unlearning in large language models is genuine by comparing the model's internal representations — hidden states and activations — before and after unlearning, rather than trusting output-level behavior alone, thereby probing whether "forgotten" knowledge has truly disappeared from within the model.

Background & Motivation

Machine unlearning for large language models aims to remove specific knowledge — such as private data, copyrighted content, or harmful information — from a trained model without retraining it from scratch. The dominant way to judge whether unlearning has succeeded is to inspect the model's outputs: if the model no longer answers questions about the target knowledge, or refuses to reproduce it, the knowledge is declared "forgotten."

However, output-level evaluation is fragile. A model may simply suppress a memorized answer at the surface while still retaining the underlying information in its internal computation. In such cases the knowledge can resurface under paraphrased prompts, adversarial probing, relearning with a handful of examples, or quantization — showing that the forgetting was cosmetic rather than genuine.

Key Challenge: Output-level unlearning metrics can be misleading. A model that appears to have "forgotten" a fact at its output may still encode that fact in its hidden representations, so surface-level refusal or accuracy drops are not reliable evidence that the knowledge has actually been erased from the model.

This motivates evaluating unlearning at the level where knowledge actually resides — the model's internal representations. If unlearning is genuine, the way the model internally represents the forgotten knowledge should meaningfully differ from how it represented that knowledge before unlearning, and ideally should become indistinguishable from how the model treats knowledge it never possessed. This work investigates whether comparing internal representations offers a more faithful signal of true forgetting than behavioral metrics alone.

Proposed Method

The study frames unlearning evaluation as a comparison problem over internal representations, contrasting the model's internal state across conditions to diagnose whether target knowledge has genuinely been removed.

Representation Extraction

For a given probe input relating to the target (forget) knowledge, hidden states and intermediate activations are extracted from the model across layers. The same extraction is performed for the model before and after unlearning, so that each condition yields a comparable internal fingerprint of how the model processes the relevant content. Extraction also covers retained knowledge inputs, so the analysis can distinguish changes specific to the forget target from incidental drift.

Comparison Across Conditions

The extracted representations are compared along two axes: (a) before vs. after unlearning for the forget knowledge, measuring how much the internal state actually shifted, and (b) forget vs. retained knowledge within the unlearned model, measuring whether the model now treats the supposedly forgotten content differently from knowledge it still holds. Similarity between representations is quantified so that genuine erasure and mere output suppression yield distinguishable patterns.

Diagnosing Genuine vs. Superficial Forgetting

By interpreting the comparison, the method assesses whether unlearning was internal or only surface-level. If the internal representation of the forget knowledge remains close to its pre-unlearning state — while the output merely refuses or errs — this indicates superficial forgetting where the knowledge is suppressed but still latent. If the representation has meaningfully changed and moved away from its original encoding, this is evidence of more genuine internal removal. This representation-level diagnosis serves as a complementary, more faithful evaluation signal alongside conventional output-based metrics.

Key Points

Beyond output behavior: Unlearning success should not be judged solely from whether the model stops producing target answers, since suppression at the output can mask retained internal knowledge.
Representations as the evaluation target: Comparing hidden states and activations before and after unlearning provides a window into whether knowledge was actually removed from the model's internal computation.
Two-way comparison: Contrasting the forget target both against its own pre-unlearning representation and against retained knowledge helps separate genuine erasure from incidental changes or cosmetic suppression.
Detecting latent knowledge: Representation-level analysis can flag cases where "forgotten" knowledge remains latent and could resurface under paraphrasing, adversarial prompting, or relearning — scenarios that output metrics may miss.
Complement to behavioral metrics: The approach is meant to augment, not replace, conventional evaluation, offering a more faithful and harder-to-game signal of true forgetting.

Why It Matters

As unlearning is increasingly proposed as a mechanism for privacy compliance, copyright removal, and safety, the way we measure its success becomes critically important. If evaluation relies only on output behavior, unlearning methods risk being credited for forgetting that is merely hidden — a false sense of safety that can break down under real-world probing.

By anchoring evaluation in internal representations, this work pushes toward a more trustworthy standard for verifying that sensitive knowledge has genuinely been removed rather than concealed. This is especially valuable for auditing and certifying unlearned models, where superficial forgetting could otherwise pass undetected.

The study also connects to a broader interpretability agenda: understanding where and how knowledge is stored in a model is a prerequisite for reliably removing it. It builds on the lab's prior work on knowledge unlearning in LLMs (EMNLP 2025), extending the focus from producing better forgetting toward faithfully evaluating whether forgetting has actually occurred.

Unlearning Interpretability