Goodfire Research 發現語言模型中的神經元羣落
Discovering Neuron Communities in Language Models
Goodfire Research 以梯度歸因及社羣檢測,從語言模型中找出反覆協作的小型神經元羣落,並在留出提示詞上檢驗其必要性與充分性。研究以 Llama-3.1-8B 和來自 FineWeb-Edu、English Wikipedia、FineMath 的 10M prompts 執行算法,並在 Qwen 3.6 35B-A3B 上複現。
Neurons are a natural starting point for understanding neural networks, but interpreting them can be difficult: an individual neuron may activate in seemingly unrelated contexts, while a single prompt activates many neurons whose contributions are hard to disentangle.
Recent evidence, however, suggests that neurons are more interpretable than previously believed: with better tools and a closer look at their computational roles, individual neurons begin to make sense. Here, we study how small groups of neurons work together to implement common model behaviors, from grammatical agreement to copying and arithmetic.
Removing these groups substantially changes the model’s predictions, while retaining them can largely preserve predictions even after ablating almost all other MLP neurons in the layers involved. Tests on held-out prompts show that these communities – often fewer than ten neurons – are both necessary and sufficient to explain the contributions of the relevant layers.
We discover these groups without specifying their functions in advance. Across millions of prompts, we use gradient-based attribution to identify influential neurons, connect neurons repeatedly attributed together in a graph, and apply community detection. We then test the resulting communities for necessity and sufficiency and examine the behaviors they support.
These results add to the evidence that neural networks already contain interpretable structure if we know how to look for it. Neuron communities can serve as computational building blocks for model behaviors, offering a level of abstraction between individual neurons and higher-order computations. Because our discovery method is unsupervised, we can find communities at scale, providing concrete starting points for agents to investigate their functions and test explanations of how they work. This offers a path from individual discoveries toward a fuller understanding of neural networks.
Background
Distributed representation offers an explanation for why neurons are hard to interpret: features may overlap across neurons, giving each neuron multiple roles. This motivates methods such as transcoders and parameter decomposition, whose interpretable units need not align with individual neurons.
Yet Arora et al. (2025) recover neuron circuits as sparse and faithful as learned-feature circuits on subject–verb agreement, and Feucht et al. (2026) identify 28 neurons implementing reusable base-10 addition. Shabalin and Piotrowski (2026) also find that many active transcoder features align with individual neurons. In our recent work (Tigges et al., 2026), we showed that models can be significantly sparsified in the neuron basis while preserving many model behaviors. Together, these results suggest that some concepts and computations can be captured by surprisingly few neurons.
Method
We want to discover groups of neurons that repeatedly work together, without deciding in advance what behaviors to look for.
ADiscover recurring neuron communities
Step 1
Neuron attribution
Score how much each neuron drives each prediction
Step 2
Build neuron graph
Link neurons that repeatedly drive the same predictions
Step 3
Discover communities
Groups of neurons that keep working together
Necessity
high KL vs. original
Ablate the community → prediction changes
Sufficiency
low KL vs. original
Ablate everything else → prediction is preserved
Step 1: Neuron attribution. We run the model on millions of prompts and use gradient-based attribution to estimate which neurons contribute most to each next-token prediction.
Step 2: Build a neuron graph. We build a graph, representing neurons that repeatedly contribute to the same predictions. Each node is a neuron, and each edge quantifies how often two neurons co-occur together, adjusted for each neuron’s individual frequency.
Step 3: Discover communities. We use graph community detection to find groups of neurons that repeatedly work together.
We then test these candidate communities through causal interventions on held-out prompts:
- Necessity: Removing the community substantially changes the model’s predictions.
- Sufficiency: Keeping the community1 largely preserves predictions even after removing the other MLP neurons in those layers.
For the results below, we run the algorithm on Llama-3.1-8B with 10M prompts from FineWeb-Edu, English Wikipedia and FineMath. We later replicated this on Qwen 3.6 35B-A3B.2
Case study: an + vowel agreement
We start with one example found by our algorithm above: a community of six neurons spread across six layers: L2:5664, L5:3846, L8:13390, L12:14182, L13:3604, and L14:11888. Below, we call a community active when it is necessary for the prediction: removing its neurons substantially changes the model’s output.
This community is active on prompts ending in the article “an.” The example prompts show a consistent pattern: the model originally predicts a word starting with a vowel. Zero-ablating these six neurons shifts its prediction toward words that still fit the context but start with a consonant. Conversely, keeping these six neurons (alongside four “always-on” neurons) while removing the remaining MLP neurons in those six layers (L2, L5, L8, L12, L13, L14) largely preserves the original prediction.

It turns out that this community is active on all “an” predictions. In the figure above we test a random set of prompts ending in the article “an,” without selecting them based on whether the community is active. We compare three interventions:
- Remove the community (orange). Removing just six neurons produces large changes in the next-token distribution, measured by KL divergence on the left, and in the probability of the model’s original top prediction on the right.
- Remove the MLP computation in the six layers (green). This produces similarly large changes.
- Keep the community within those layers (purple). Removing the other MLP neurons while retaining the community largely preserves both the prediction distribution and the original top token’s probability.
It is worth noting that our sufficiency claims are limited to the layers which contain community neurons, in this case six of them. The rest of the model still contributes to the prediction. These tests establish that the relevant computation in the tested layers is neuron-sparse, but don’t say anything about how the community interacts with the rest of the model and exactly what it computes to favor vowel-starting words.
Further examples
The “an” community is one of thousands discovered by our method. The interactive examples above show the range of computations these groups support, together with the model’s predictions before and after removing each community.
- Grammar. 17 neurons across eight layers help select “is” or “are” after “there.” Removing them changes the continuation of “Ten cards are put into five envelopes so that there” from “are” to “is.”
- Retrieval and computation. For instance, community supporting unit conversion and more generic multiplication. Some communities assist with in-context retrieval: e.g. seven neurons across four layers help retrieve names mentioned earlier in a prompt.
- Topic. Some communities are active on a specific topic, e.g. 3-neuron community activates on different trigonometric functions or 6-neuron community that activates on terms related to elevation.
- Formatting. These communities encode formatting rules and conventions, e.g. 7-neuron community responsible for continuing with ALL CAPS in a phrase.
- Common phrases. Many communities encode common phrases or a specific use of a word, e.g. 37-neuron community that’s active when predicting ‘out’ as a part of a phrasal verb (carry out, find out, figure out)
- Token suppression. This is one of the most common communities by type. Usually 1-2 neurons suppress one specific word or token.
You can explore hundreds more communities in our neuron community explorer.
The discovered communities also reveal patterns in where different computations occur. The heatmap shows how each category’s neurons are distributed across layers. Grammar-related neurons are more concentrated from layer 18 onward, while retrieval and computation—including arithmetic—peak around layers 17–22. Token suppression is concentrated in the final layers. These patterns suggest that neuron communities can help us map not only individual computations, but also how different kinds of computation are organized across the model’s depth.
Conclusion
Prior work has shown that individual neurons can be interpretable. In this work, we show that neurons form small groups that act as reusable units of computation, supporting behaviors from grammatical agreement to copying and arithmetic. We can discover these groups at scale without specifying their functions in advance, then test whether they are necessary and sufficient for predictions within selected layers.
These communities offer a level of abstraction between individual neurons and model behaviors, helping us investigate how neurons combine to implement higher-order computations. The explanation remains partial: many predictions remain unexplained, and identifying a necessary and sufficient community does not yet tell us exactly how its neurons compute or interact with the rest of the model. But unsupervised discovery gives us concrete starting points for investigating these questions at scale, including through agents that interpret communities and test hypotheses about their mechanisms. Connecting these findings could help us build a fuller understanding of how neural networks work—and how to control their behavior.
Citation
To cite this work, please cite the paper.
Footnotes
-
We also identify 17 “always-on” neurons across the model’s 32 layers and keep them on for all sufficiency tests. ↩
-
We identified hundreds of thousands of these neuron communities in Qwen 3.6 35B-A3B as well, including many circuits that encode pieces of knowledge—e.g., state capitals or famous painters associated with post-Impressionism. ↩
來源:Goodfire Research · goodfire.com