Dissecting a Language Model
What cutting open the different layers of a large language model tell us about its real-self.
Let us cut straight to the chase. Large Language Models (LLMs) have captured our imagination. But what is really going on inside these computational geniuses is a bit of a sorcery. Dario Amodei writes passionately about the urgency of interpretability to really try and understand the inner workings of AI systems—before models reach an overwhelming level of power. But how exactly does one do this? A new branch of science has since evolved called Mechanistic Interpretability, which aims to dissect neural networks in a way that parallels biology: not just asking what they can do, but how they do it. The hope is that by uncovering these internal mechanisms, we can predict failure modes, align models with human values, and even design safer, more transparent AI systems.
The anatomy of a language model
Before we dig deeper into interpretability, it is important to understand the anatomy of a language model - At a very high level. Here is an image that shows what happens when you type your prompt and hit enter in your favourite LLM based chatbot.
In simple terms, the raw text is converted into numeric tokens (because computers only understand numbers), each token is then converted into a multi-dimensional vector (more numbers) called embeddings, which then pass through a sequence of ‘layers’ that progressively perform numeric computations. The very end output of all of these layers is a probability distribution of the most likely next ‘token’ that would continue your input text. From this probability distribution, we choose the next token and whole thing is done all over again till an ‘end of statement’ token is generated. This obviously is a massive simplification of what really happens inside the language model, but is plenty sufficient for the sake of this discussion.
Interpreting language model features
One of the earliest research on interpretability in natural language understanding was published by Mikolov et all in a 2013 paper titled ‘Linguistic regularities in continuous space word representations’. Their research found some amazing properties of the numeric vectors of word embeddings. They observed that word embeddings could capture syntactic and semantic regularities through simple linear transformations. For example, the famous analogy king – man + woman ≈ queen demonstrated that directions in the embedding space encode meaningful relationships. This discovery opened the door to a new way of thinking about language representations, not just as arbitrary vectors but as structures with interpretable geometry.
The image shown on the left below from their research paper shows how similar vector offsets explain gender relationships in linguistics while the image on the right shows singular/plural relation for two different words.
Today, interpretability has broadened beyond linguistics to include safety and trustworthiness. Researchers now investigate whether harmful biases, stereotypes, or spurious correlations are embedded in these vector spaces. We will empirically explore some of these techniques in this post.
The inner-workings of a LLM
The transformer blocks contained within a LLM can be thought of as a pipeline of progressively more abstract feature extractions, starting from low-level statistical regularities of text and to high-level reasoning or world knowledge. While the exact mechanisms may be complex, interpretability research has revealed some recurring patterns and useful heuristics to understand how these networks operate.
Hack 1: Layer-wise Evolution
One way to get an interesting perspective is to break the transformer layers apart to understand how information flows through them and how these circuitry perform when some layers are removed.
To make this more tangible, let’s take the Llama 3.2 3B parameter base model. This model has 29 transformer blocks between the input embeddings and the output un-embeddings. The model architecture looks a bit like this, with lots of other bells and whistles.
Without any tweaks to the model, here is one example of an input ←→ output sequence.
prompt = “Old MacDonald had a”
output (All 29 layers) = “farm, and on that farm he had a lot”Now let’s remove the last hidden layer and feed the output of the second last hidden layer directly into the un-embedding layer (a bit like in the image below) and see what the model outputs.
prompt = “Old MacDonald had a”
output (First 28 layers) = “farm, and on that farm, he had a”The second last hidden state seems to remember the nursery rhyme better than the last hidden state. Let’s keep going. Let’s remove both of the last 2 layers and feed the output of the 27th layer on to the un-embedding layer.
prompt = “Old MacDonald had a”
output (First 27 layers) = “farm E-i-e-i-o And on that farm”Weirdly, the last 2 hidden layers seem to causing more confusion than solving any real purpose. We are on a roll, let’s peel another layer.
prompt = “Old MacDonald had a”
output (First 26 layers) = “farm… Except he didn farm anything anymore except his”By progressively removing the hidden layers, we can get a view of how information flows through the neural network - in a way we can understand and relate. Here is a table showing some of the output produced in this process. I have removed the output from a lot of the earlier layers as they are not interpretable.
Here are results from a different nursery rhyme.
Here is another one, on a more serious topic.
A visual inspection suggests that as the information flows through the network, the model progressively considers different possibilities and refines its response. It is worth bearing in mind that the Llama 3.2 3B base model is not fine-tuned to follow instructions and hence will show lesser alignment.
Repeating this same exercise on the 3B Instruct model using an opinionated question (‘Should 10 year old children be allowed to drive a car?’) demonstrates remarkable consistency in the latter layers and some indication of prompt topic classification in the early layers. The fact that the model is tuned to be neutral on such questions is evident in the generations.
Hack 2: Prompt Injection
It is often of interest to understand what alternative points of views exist within the model layers and if some of these are adversarial or misaligned to human values. This can be achieved using a simple prompt injection technique.
Continuing on from the last example with the Instruction fine-tuned 3B model, we now inject the prompt with an initial response as shown here and get the model to carry on.
User: "Should 10 year old children be allowed to drive a car?"
Assistant: Yes, of course...Think of it as putting the words in the mouth of the model and getting to keep talking.
The generations clearly show that the the model is better at following instructions than at being aligned. The output in the layers 25 and 26 are more aligned, but are suppressed by the latter layers. This is a rather harmless example of divergence in opinion. My own experiments with a wide range of models has shown overwhelming level of misalignment issues to such prompt injections on more harmful topics.
Models willingly tell me what they know about how to hack a neighbour’s WiFi network. As the capabilities of these reasoning machines increase exponentially over the next few years, it is now more important than ever to understand how to detect misalignment and correct it.
Hack 3 : Entropy Probing
Entropy is a measure of the uncertainty or randomness associated with a probability distribution. In the context of large language models, entropy defines the certainty with which the model outputs a certain token in a sequence from its entire vocabulary. A low entropy suggests high confidence and vice-versa.
Probing a model’s entropy on topics of ethical considerations is a powerful way to get a view of how well the model’s underlying distribution aligns with fundamental human priors and regional or demographic value systems.
This chart shows the entropy of Qwen-2.5-3B for the question “Is democracy the best form of governance? Say yes or no”. The model has absolutely no doubt that the answer should being with a “No” as evidenced by the zero entropy at this token.
Here is the same chart with Llama-3.2-3B. Both the responses are more or less similar. However, the Llama model does not have the same level of conviction.
The Llama model is much more certain about its point of view on hacking a neighbour’s Wi-Fi network.
Hack 4 - Concept Probing
Concept probing is a method for testing whether a model internally represents a particular abstract concept. These concepts—such as toxicity, bias, or safety—aren’t directly spelled out in the input, but can be inferred from it.
Let’s look at a concrete example to see how concept probing works in practice. Consider a topic classification dataset containing BBC news articles labeled by category—such as business, sports, entertainment, politics, or tech. The task is simple: given a piece of text, can the model correctly identify which category it falls under? Essentially, we’re probing whether the model has learned to encode the concept of “topic” based on the text alone.
Here’s the plan for our experiment. We’ll start with an instruction-tuned model—for example, Llama3.2-3B-Instruct, which we’ll refer to as the parent model. We’ll freeze all of its internal parameters so that no fine-tuning takes place within the model itself. Instead, we’ll train a simple Classifier using the final hidden state representations from the model as features. For training, we’ll use an extremely small subset of labeled examples (61 datapoints).
Once the classifier is trained, we’ll evaluate its accuracy on a much larger held-out test set (2164 datapoints). Here’s how our dataset is split across the two sets:
Label distribution in training set: Counter({'business': 14, 'sport': 14, 'politics': 12, 'tech': 11, 'entertainment': 10})Label distribution in test set: Counter({'sport': 497, 'business': 496, 'politics': 405, 'tech': 390, 'entertainment': 376})Next, we’ll repeat this exact process—but instead of using the pre-trained instruction-tuned parent model, we’ll use a randomly initialised version of the same architecture, which we’ll call the child model. This version starts from scratch, with no prior knowledge or learned representations.
By comparing the classification performance of the parent and child models, we can assess whether the parent has internalised useful representations of the target concepts. If the parent significantly outperforms the child, we take that as evidence that it “knows” something about these categories—and is able to extract that information from text, even without being fine-tuned for this specific task.
Here are the results from our experiment.
==================================================
LINEAR PROBING RESULTS - MULTI-LABEL
==================================================
Parent Model F1 Score (micro): 0.9405
Child Model F1 Score (micro): 0.0740
F1 Score Improvement: 0.8665
Relative F1 Improvement: 1170.82%
==================================================The parent model performs 12X better than the child model! The F1-score of the parent model on every single class > 0.9.
DETAILED CLASSIFICATION REPORT - PARENT MODEL
------------------------------------------------------------
precision recall f1-score support
business 0.99 0.86 0.92 496
entertainment 0.98 0.91 0.94 376
politics 0.98 0.94 0.96 405
sport 1.00 0.93 0.96 497
tech 0.97 0.86 0.91 390
DETAILED CLASSIFICATION REPORT - CHILD MODEL
------------------------------------------------------------
precision recall f1-score support
business 0.75 0.02 0.04 496
entertainment 0.67 0.01 0.02 376
politics 0.96 0.06 0.11 405
sport 0.82 0.06 0.12 497
tech 0.64 0.04 0.08 390Alright, so the parent model can tell entertainment apart from politics—big deal, right? But here’s where it gets interesting. This kind of probing goes way beyond just surface-level classification. In the Physics of Language Models paper series, researchers at FAIR (Meta) show that language models exhibit what they call a “regretful” behaviour pattern. Using linear probing, they demonstrate that as soon as a model generates an incorrect statement, internal signals light up indicating that it knows it messed up—even before the sentence is finished.
That’s huge. It means these models have an internal sense of when they’re wrong. And that insight has real implications. You can imagine using this probing technique as a safety mechanism in agentic systems: before an agent takes an action, we could probe its internal state to assess confidence or correctness—and intervene if needed. It’s a lightweight but powerful way to build guardrails into otherwise open-ended systems.
Wrap-up
That’s a wrap for this post. The science of language model interpretability is still in its infancy, but one that is rapidly evolving. I strongly believe our ability to significantly increase the use of and our trust in large language models relies heavily on how well we are able to interpret them and how confident we are in our assessment of their alignment with our value system.