One-Line Summary
A comparative study of the basic representation units used to analyze knowledge stored inside large language models — individual neurons versus features extracted by Sparse Autoencoders (SAEs) — examining which unit provides more interpretable, disentangled, and faithful access to how factual and linguistic knowledge is encoded within the model.
Background & Motivation
Understanding where and how knowledge lives inside a large language model is a central goal of mechanistic interpretability. To probe, localize, or edit the knowledge a model has memorized, researchers must first choose an atomic unit of analysis: the smallest component whose activation is inspected, attributed to a piece of knowledge, and potentially intervened upon. The two dominant candidates today are the model's own individual neurons (dimensions of hidden activations) and the features recovered by Sparse Autoencoders (SAEs) trained on those activations.
Neurons are the native, readily available unit — no extra training is required, and prior work on "knowledge neurons" has attributed specific facts and linguistic categories to small groups of them. However, neurons are widely observed to be polysemantic: a single neuron often fires for many unrelated concepts because the model packs more features into its representation space than it has dimensions (superposition). This entanglement makes neuron-level attribution ambiguous and can produce a misleading picture of how knowledge is organized.
SAEs were introduced to address exactly this problem. By reconstructing hidden activations as a sparse linear combination of an over-complete dictionary of features, SAEs aim to disentangle superposed concepts into monosemantic units, each ideally corresponding to a single, human-interpretable concept. Yet SAE features come at a cost: they require additional training, introduce reconstruction error, and their faithfulness to the model's actual computation is itself under scrutiny.
Key Tension — Polysemantic Neurons vs. Sparse Features:
- Neurons are native but entangled: They are directly part of the model's computation, but polysemanticity and superposition mean one neuron mixes many concepts, blurring knowledge attribution.
- SAE features are disentangled but derived: They promise clean, monosemantic units that separate distinct pieces of knowledge, but they are learned approximations layered on top of the model, raising questions of faithfulness and reconstruction fidelity.
- The open question: When the goal is to analyze knowledge stored in an LLM, which unit yields more interpretable, disentangled, and faithful conclusions — and under what conditions?
This work situates itself within the lab's broader interpretability line, which asks how internal representations mediate knowledge access and retrieval in LLMs (e.g., Query Lens, ICML 2026). Rather than assuming a single "right" unit, it treats the choice of representation unit as an empirical question and systematically compares neurons and SAE features on the same knowledge-analysis tasks.
Two Views of the Same Representation
| Aspect | Neurons | SAE Features |
|---|
| Origin | Native hidden dimensions of the model | Learned dictionary trained on activations |
| Extra training? | None required | SAE must be trained per layer/site |
| Interpretability | Often polysemantic (mixed concepts) | Aims for monosemantic (single concept) |
| Disentanglement | Limited by superposition | Explicitly encourages sparse separation |
| Faithfulness concern | Directly on the computation path | Reconstruction error may distort signal |
| Cost / convenience | Low cost, immediately available | Higher cost, dictionary size to tune |
Proposed Method
The study frames a controlled, head-to-head comparison in which neurons and SAE features are evaluated as interchangeable analysis units for the same knowledge-probing questions. The design isolates the effect of the unit choice from the underlying knowledge being studied.
1
Define a Common Knowledge-Analysis Setting
A shared set of knowledge probes is fixed — factual associations and linguistic properties that the model has plausibly memorized — so that both candidate units are tested against identical targets. Fixing the knowledge under study ensures that any difference in the resulting analysis is attributable to the representation unit rather than to a change in the task.
2
Extract Both Candidate Units from the Same Activations
For a chosen model site (e.g., hidden activations at a given layer), the neuron view uses the raw activation dimensions directly, while the SAE feature view passes the same activations through a trained Sparse Autoencoder to obtain a sparse, over-complete feature basis. Both views therefore describe the same underlying representation, differing only in how it is decomposed into units.
3
Localize Knowledge to Units
For each piece of knowledge, the most responsible units are identified in both spaces — the neurons most strongly associated with the target, and the SAE features most strongly associated with the target. This yields, for every probe, a neuron-based localization and an SAE-based localization that can be directly compared for how concentrated, clean, and consistent the attribution is.
4
Assess Interpretability and Disentanglement
Each identified unit is examined for what concepts it responds to. The analysis asks whether a unit corresponds to a single coherent concept (monosemantic) or mixes several unrelated ones (polysemantic), and whether distinct pieces of knowledge map to distinct units rather than collapsing onto shared, overloaded units. This captures the practical interpretability and disentanglement of each representation unit.
5
Test Faithfulness via Intervention
To move beyond correlation, the identified units are perturbed — suppressing or amplifying the neurons versus the SAE features associated with a given target — and the effect on the model's expression of that knowledge is observed. A unit is deemed faithful to the extent that intervening on it selectively changes the intended knowledge without broadly disrupting unrelated behavior, providing a causal rather than merely descriptive comparison.
Controlled Comparison Principle:
Because both unit types are derived from the same activations and evaluated on the same probes, differences in interpretability, disentanglement, and faithfulness can be attributed to the choice of representation unit itself — making the comparison a clean test of "neurons vs. SAEs" rather than a confound of different tasks or models.
Key Points
- Unit choice is consequential: The conclusions one draws about how knowledge is stored in an LLM depend substantially on whether neurons or SAE features are used as the unit of analysis; the two views are not interchangeable.
- Neurons are convenient but entangled: Individual neurons are immediately available and require no extra training, but their polysemantic nature and superposition make knowledge attribution ambiguous and harder to interpret cleanly.
- SAE features favor disentanglement: Sparse Autoencoder features tend to separate superposed concepts into more monosemantic units, offering cleaner and more human-interpretable localization of individual pieces of knowledge.
- Faithfulness is the crucial test: Interpretability alone is not enough — a unit is only useful for knowledge analysis if intervening on it selectively and reliably alters the targeted knowledge, so faithfulness (via intervention) is treated as a first-class criterion.
- Trade-offs, not a clear winner: The comparison surfaces a trade-off — SAE features gain interpretability and disentanglement at the cost of extra training and reconstruction error, while neurons offer directness and lower cost at the cost of entanglement.
- Guidance for interpretability practice: The findings inform which representation unit to reach for depending on the goal, whether the priority is clean interpretation, causal editing, or low-overhead analysis.
Why It Matters
The choice of representation unit is a foundational, often unexamined assumption underlying much of interpretability research. By making that choice the explicit object of study, this work clarifies what is gained and lost when analyzing LLM knowledge through neurons versus SAE features.
- Puts a methodological decision under the microscope: Many interpretability findings implicitly depend on the unit of analysis; this study makes that dependence visible and testable, encouraging more careful reporting of how knowledge localization is performed.
- Bridges convenience and interpretability: It weighs the practical appeal of native neurons against the disentanglement benefits of SAEs, giving practitioners a principled basis for choosing between them rather than defaulting to whichever is easier.
- Strengthens knowledge editing and control: By emphasizing faithfulness via intervention, the work connects representation analysis to downstream goals such as knowledge editing, unlearning, and steering — where the unit chosen directly determines how precisely a model's behavior can be modified.
- Extends the lab's interpretability agenda: The study complements the lab's work on how internal representations mediate knowledge access and retrieval (e.g., Query Lens, ICML 2026), contributing to a coherent line of research on making the internal knowledge of LLMs legible and controllable.
Broader Insight: Before we can answer "how does an LLM store this knowledge?", we must decide in terms of what we describe that storage. Neurons and SAE features offer two different lenses on the same representation, and each shapes the conclusions we can draw. Treating the representation unit as an empirical choice — rather than a given — is a prerequisite for trustworthy, reproducible interpretability of knowledge in large language models.
Interpretability
Representation Learning