跳到正文
Goodfire Research·· 6 天前AI 評分57

Goodfire Research 發現語言模型中的神經元羣落

Discovering Neuron Communities in Language Models

AI 導讀

Goodfire Research 以梯度歸因及社羣檢測,從語言模型中找出反覆協作的小型神經元羣落,並在留出提示詞上檢驗其必要性與充分性。研究以 Llama-3.1-8B 和來自 FineWeb-Edu、English Wikipedia、FineMath 的 10M prompts 執行算法,並在 Qwen 3.6 35B-A3B 上複現。

正文

Neurons are a natural starting point for understanding neural networks, but interpreting them can be difficult: an individual neuron may activate in seemingly unrelated contexts, while a single prompt activates many neurons whose contributions are hard to disentangle.

Recent evidence, however, suggests that neurons are more interpretable than previously believed: with better tools and a closer look at their computational roles, individual neurons begin to make sense. Here, we study how small groups of neurons work together to implement common model behaviors, from grammatical agreement to copying and arithmetic.

Removing these groups substantially changes the model’s predictions, while retaining them can largely preserve predictions even after ablating almost all other MLP neurons in the layers involved. Tests on held-out prompts show that these communities – often fewer than ten neurons – are both necessary and sufficient to explain the contributions of the relevant layers.

We discover these groups without specifying their functions in advance. Across millions of prompts, we use gradient-based attribution to identify influential neurons, connect neurons repeatedly attributed together in a graph, and apply community detection. We then test the resulting communities for necessity and sufficiency and examine the behaviors they support.

These results add to the evidence that neural networks already contain interpretable structure if we know how to look for it. Neuron communities can serve as computational building blocks for model behaviors, offering a level of abstraction between individual neurons and higher-order computations. Because our discovery method is unsupervised, we can find communities at scale, providing concrete starting points for agents to investigate their functions and test explanations of how they work. This offers a path from individual discoveries toward a fuller understanding of neural networks.

Background

Distributed representation offers an explanation for why neurons are hard to interpret: features may overlap across neurons, giving each neuron multiple roles. This motivates methods such as transcoders and parameter decomposition, whose interpretable units need not align with individual neurons.

Yet Arora et al. (2025) recover neuron circuits as sparse and faithful as learned-feature circuits on subject–verb agreement, and Feucht et al. (2026) identify 28 neurons implementing reusable base-10 addition. Shabalin and Piotrowski (2026) also find that many active transcoder features align with individual neurons. In our recent work (Tigges et al., 2026), we showed that models can be significantly sparsified in the neuron basis while preserving many model behaviors. Together, these results suggest that some concepts and computations can be captured by surprisingly few neurons.

Method

We want to discover groups of neurons that repeatedly work together, without deciding in advance what behaviors to look for.

ADiscover recurring neuron communities

Step 1

Neuron attribution

Score how much each neuron drives each prediction

Attribution matrix of prompts by neurons, with dark cells marking neurons that strongly drive each prompt’s prediction.

Step 2

Build neuron graph

Link neurons that repeatedly drive the same predictions

Graph of neurons a to i, with thicker edges between neurons that are often attributed together.

Step 3

Discover communities

Groups of neurons that keep working together

The same graph split into two communities: neurons a, c, f and g in orange, and d, e, h and i in blue.

Necessity

high KL vs. original

Ablate the community → prediction changes

Grid of neurons by layers in which only the community neurons are ablated.

Sufficiency

low KL vs. original

Ablate everything else → prediction is preserved

Grid of neurons by layers in which every neuron except the community is ablated in the tested layers.

Step 1: Neuron attribution. We run the model on millions of prompts and use gradient-based attribution to estimate which neurons contribute most to each next-token prediction.

Step 2: Build a neuron graph. We build a graph, representing neurons that repeatedly contribute to the same predictions. Each node is a neuron, and each edge quantifies how often two neurons co-occur together, adjusted for each neuron’s individual frequency.

Step 3: Discover communities. We use graph community detection to find groups of neurons that repeatedly work together.

We then test these candidate communities through causal interventions on held-out prompts:

  • Necessity: Removing the community substantially changes the model’s predictions.
  • Sufficiency: Keeping the community1 largely preserves predictions even after removing the other MLP neurons in those layers.

For the results below, we run the algorithm on Llama-3.1-8B with 10M prompts from FineWeb-Edu, English Wikipedia and FineMath. We later replicated this on Qwen 3.6 35B-A3B.2

Case study: an + vowel agreement

We start with one example found by our algorithm above: a community of six neurons spread across six layers: L2:5664, L5:3846, L8:13390, L12:14182, L13:3604, and L14:11888. Below, we call a community active when it is necessary for the prediction: removing its neurons substantially changes the model’s output.

This community is active on prompts ending in the article “an.” The example prompts show a consistent pattern: the model originally predicts a word starting with a vowel. Zero-ablating these six neurons shifts its prediction toward words that still fit the context but start with a consonant. Conversely, keeping these six neurons (alongside four “always-on” neurons) while removing the remaining MLP neurons in those six layers (L2, L5, L8, L12, L13, L14) largely preserves the original prediction.

Distribution of KL divergence from the unmodified model on prompts ending in “an”: removing the six community neurons or all MLP neurons in their layers causes large changes, while retaining the community largely preserves predictions.Mean absolute change in the original top-token probability: 0.42 when removing the six community neurons, 0.38 when removing all MLP neurons in their six layers, and 0.08 when retaining the community and always-on neurons in those layers.

On random prompts ending in "an", removing the six community neurons (orange) changes predictions almost as much as removing all MLP neurons in the six layers (green). Keeping the six community neurons and the four always-on neurons in those layers (purple) largely preserves predictions. Left: KL divergence from the unmodified model. Right: mean absolute change in its original top-token probability.

It turns out that this community is active on all “an” predictions. In the figure above we test a random set of prompts ending in the article “an,” without selecting them based on whether the community is active. We compare three interventions:

  • Remove the community (orange). Removing just six neurons produces large changes in the next-token distribution, measured by KL divergence on the left, and in the probability of the model’s original top prediction on the right.
  • Remove the MLP computation in the six layers (green). This produces similarly large changes.
  • Keep the community within those layers (purple). Removing the other MLP neurons while retaining the community largely preserves both the prediction distribution and the original top token’s probability.

It is worth noting that our sufficiency claims are limited to the layers which contain community neurons, in this case six of them. The rest of the model still contributes to the prediction. These tests establish that the relevant computation in the tested layers is neuron-sparse, but don’t say anything about how the community interacts with the rest of the model and exactly what it computes to favor vowel-starting words.

Further examples

The “an” community is one of thousands discovered by our method. The interactive examples above show the range of computations these groups support, together with the model’s predictions before and after removing each community.

  • Grammar. 17 neurons across eight layers help select “is” or “are” after “there.” Removing them changes the continuation of “Ten cards are put into five envelopes so that there” from “are” to “is.”
  • Retrieval and computation. For instance, community supporting unit conversion and more generic multiplication. Some communities assist with in-context retrieval: e.g. seven neurons across four layers help retrieve names mentioned earlier in a prompt.
  • Topic. Some communities are active on a specific topic, e.g. 3-neuron community activates on different trigonometric functions or 6-neuron community that activates on terms related to elevation.
  • Formatting. These communities encode formatting rules and conventions, e.g. 7-neuron community responsible for continuing with ALL CAPS in a phrase.
  • Common phrases. Many communities encode common phrases or a specific use of a word, e.g. 37-neuron community that’s active when predicting ‘out’ as a part of a phrasal verb (carry out, find out, figure out)
  • Token suppression. This is one of the most common communities by type. Usually 1-2 neurons suppress one specific word or token.

You can explore hundreds more communities in our neuron community explorer.

Heatmap of how each community category’s neurons are distributed across the model’s layers.
Distribution of community neurons across model layers, grouped by behaviour. Each cell shows the percentage of a category’s neurons in that layer, with darker shades indicating a larger share. Token suppression is concentrated in the final layers, while other behaviours span a broader range of layers.

The discovered communities also reveal patterns in where different computations occur. The heatmap shows how each category’s neurons are distributed across layers. Grammar-related neurons are more concentrated from layer 18 onward, while retrieval and computation—including arithmetic—peak around layers 17–22. Token suppression is concentrated in the final layers. These patterns suggest that neuron communities can help us map not only individual computations, but also how different kinds of computation are organized across the model’s depth.

Conclusion

Prior work has shown that individual neurons can be interpretable. In this work, we show that neurons form small groups that act as reusable units of computation, supporting behaviors from grammatical agreement to copying and arithmetic. We can discover these groups at scale without specifying their functions in advance, then test whether they are necessary and sufficient for predictions within selected layers.

These communities offer a level of abstraction between individual neurons and model behaviors, helping us investigate how neurons combine to implement higher-order computations. The explanation remains partial: many predictions remain unexplained, and identifying a necessary and sufficient community does not yet tell us exactly how its neurons compute or interact with the rest of the model. But unsupervised discovery gives us concrete starting points for investigating these questions at scale, including through agents that interpret communities and test hypotheses about their mechanisms. Connecting these findings could help us build a fuller understanding of how neural networks work—and how to control their behavior.

Citation

To cite this work, please cite the paper.

Footnotes

  1. We also identify 17 “always-on” neurons across the model’s 32 layers and keep them on for all sufficiency tests. ↩

  2. We identified hundreds of thousands of these neuron communities in Qwen 3.6 35B-A3B as well, including many circuits that encode pieces of knowledge—e.g., state capitals or famous painters associated with post-Impressionism. ↩

來源:Goodfire Research · goodfire.com