Haven’t you ever wondered how an LLM is actually able to ‘think’?
I’m not talking about the pipeline, the whole sequential thing of breaking a sentence into tokens, projecting those tokens into vector embeddings, running them through multi-head attention layers and feed-forward blocks until a probability distribution over the next token falls out the other side. That stuff is well documented in the paper from Vaswani and friends in 2017 and about forty thousand blog posts written by people like me, who clearly needed something to do on a Sunday afternoon. No, I’m talking about a more ‘meta level’ such as ‘this thing is spectacularly smart, smarter than almost anything else we have built, and we do not actually understand how it works in any detail that would let us look inside and fix a specific thing we do not like’. Yes, that kind of stuff. We can observe inputs and outputs. Buuut, the interior is another matter.
In my work at the frontier of unsexy AI, you do not spend much time worrying your pretty little head about that sort of thing. We are the kings of applied AI. We like to leave the mechanistic interpretability research to the genuinely cool people at Anthropic and DeepMind who have PhDs in mathematics and office plants that are actually alive. My job is to deploy these things into factories and compliance workflows and procurement chains, convince people that the ROI is real (it is, by the way, just roughly a quarter of what the vendors claims), and figure out why the darn thing hallucinated a supplier’s bank account number into a payment instruction at two in the afternoon on a Tuesday.
Something like that last one is what got me interested in the inner world.
When you are running agentic AI in production, the governance question is kinda urgent and quite specific. I am a big believer in pre-inference and in-agent governance rather than post-hoc governance. It is less expensive and also less prone to error, but it is harder to implement, so it is not for everyone, and it requires you to actually understand something about your business and your processes but also what the model is doing internally instead of simply wrapping it in an API call and hoping for the best. My first proof of concept for the Ontological Compliance Gateway*, the OCG is now running as a business pilot in our programs, and the basic idea is that instead of checking what the agent did after it did it – the post-hoc thing – you limit where the agent can go in the first place. Pre-emptive, geometry-based governance.
And that requires knowing something about, yes, the geometry of the thing.
I hate analogies, but in case of things like space-time and a multi-dimensional latent-space, the only thing you have at your disposal is actually an analogy to describe what’s going on inside an LLM’s brain. So I think of an AI’s thinking process as a small airplane flying through a three-dimensional space filled with mountains and valleys and the occasional village. The villages are attractors with stable regions that the system naturally gravitates toward, where the plane wants to land because the terrain pulls it in. And so, most of the time the airplane lands in the right village. But sometimes it lands in a mirage we call a false attractor, and the output that comes out is wrong, fluent and always confident, but completely and utterly detached from reality. That is what we call a hallucination. And my argument, which I will develop at some length in this piece (because why not), is that if you can fix the flight path ahead of time using an ontology, then you can build guardrails – the flight corridor – that keep the plane within a bounded region of that space, and for agentic AI specifically, where the degrees of freedom are enormous and the consequences of landing in the wrong village can cascade through a workflow, that bounded latent space is kinda a must-have.
So I wanted to take this opportunity to talk about this inner world of AI, explain what is actually happening inside these systems in terms that a first-year student without a computer science background can follow without being patronized, and connect it to why any of this matters for companies deploying these things at scale. The research I have been drawing on comes from my own Neural Mind Project, a 25-chapter deep-dive across over a hundred papers on the internal cognition of large language models that I drew from the literature on the research hub ATLAS, as well as our OCG preprint which you can find on ArXiv and our latest paper “forecasting agent failure in long-horizon AI systems: trajectory prediction as a governance primitive”.
I will point to all of it throughout.
Fair warning, this is a long article. I teach machine learning and when I find something interesting I have a habit of mansplaining it until my students either understand it or fall asleep. The difference between a classroom and a blog is that you can close the tab, which I accept as the natural selection for this sort of long-form content that I regurgitate on a weekly basis.
*_If you want to know more about ontologies, read this accessible piece: Generative models guess, ontologies clean up the mess | LinkedIn
We built a thing we cannot read
Lone Skum can autonomously land rockets back on drone ship in the middle of the ocean, we now can sequence a human genome in a day, predict protein folding structures that stumped biochemists for fifty years, and yet we still can’t even tell you with what a single neuron in GPT-4 actually does or why Claude said what it said in a response to the specific prompt you gave it.
This is not that we haven’t tried though. Something we call ‘mechanistic interpretability’ is an entire research field with people that are dedicated to reverse-engineering neural networks at the level of internal mechanism. I’m not talking about the people sitting behind New Relic, observing that the input went in and the output came out, but tracing the actual computational steps that connected them. By the way, Anthropic has a substantial interpretability team, DeepMind has one and there are academic groups at MIT, Stanford and Berkeley full of people with big foreheads trying to understand the way LLMs think. And the field is producing genuinely impressive results, but at the same time it’s also producing the same feeling I get every time I look at the face of my Weiner (dog) and try to understand his internal wiring when he decided to do something else than I had in mind. Which is what happens every day.
The main reason we built something that we cannot read is actually that we did not design the thing from the ground up. We tought it as part of a training process, that essentially is a very large optimization problem, and we ran it on very large amounts of data using very large computers, and what came out of the business end, was a set of billions or nowadays even trillions of numerical weights that result in something we call ‘intelligence’. In the early days of AI, with symbolic AI, people wrote the instructions in the form of logic, but with a neural network, the instructions emerge and those things are notoriously difficult to reverse-engineer because there is simply no blueprint or a design document behind the specific configuration of any of those weights.
The comparison I use in my lectures is something like this.
If you want to understand how a calculator works, you look at the circuit diagram. There is a human-designed circuit diagram. It exists somewhere and you can trace the logic. But when you want to understand how the human brain produces the thought “I should eat something,” you are in dear trouble, because our jelly-like cauliflower was not designed by a human with a circuit diagram, it has been shaped by gazillion years of evolution into an architecture of eighty-six billion neurons with roughly a hundred trillion connections, and the best tools we have for reading it are still fundamentally indirect. Stuff like an MRI shows how blood flows, an electrode records voltage spikes in your brain, but neither one of ‘em tells you what a neuron is thinking.
And LLMs are closer to brains than calculators.
Where thoughts actually live
This is where I need to introduce a piece of mathematics. It may sound intimidating but it’s actually not that bad once you sit with it for a minute.
So, give it a minute.
In our latest paper, we formalized the idea that an LLM’s internal state at any given moment during inference can be represented as a point on what mathematicians call a Riemannian manifold.
Please don’t run.
A manifold is just a space that looks flat up close but can be curved at larger scales.
Take the surface of the Earth, that is a manifold. If you stand in a soggy field in the Netherlands (big chance you are), and where the terrain conspires to make the concept of hills feel theoretical, the ground looks flat in every direction. That is why us Dutchies use our bicycles so much, but when you zoom out to the size of the planet, it is obviously curved. Now, that is a RM. The point is that locally flat does not mean globally flat, and the geometry of the large-scale structure matters enormously for understanding how things move through the space.
The internal state of an LLM is a very high-dimensional space. Llama 3 (may it rest in piece), with 70 billion parameters has hidden state dimensions in the tens of thousands. Each position in that space corresponds to some configuration of the model’s internal representations, with things like what it currently “believes” about the context and what concepts are activated or what information has been integrated from earlier in the sequence, and as the model generates each token, that state moves. The sequence of positions the state traces through this space during a generation is what we, in our paper, call ‘ze trajectory’ (pronounce in French – dunno why – just give it a shot).
The geometry of this space is not random.
Researchers that were studying the flow of information inside a transformer model found that related ideas tend to be stored close together. Grammatical concepts often appear in specific directions, and factual knowledge forms recognizable patterns.
After each processing step inside the model, something interesting happens. The model updates its internal “best guess” about what is going on in the text it is reading. Say you’re reading a book, a crime novel, one page at a time, and you’re constantly revising your theory about who dunnit when you read the next page and new information comes in. Shai and his academic friends found that this theory has a real, measurable structure inside a model. The ‘page updates’ are not scattered randomly across billions of numbers, but they appear to be organized in a consistent and quite predictable pattern that researchers can actually map and study. Every layer (an attention mechanism plus two broad layers of neurons) makes the internal theory of ‘who dunnit’ a little more refined, until by the end the model has a complete working picture of the input and is ready to produce an output.
Now, this matters for governance because the model’s internal state can now be treated like a point in a space with semantic meaning, and that point is moving in ways that reflect what the model is computing.
This means, that if you can characterize the region of the space that corresponds to accurate, compliant, ontologically valid reasoning, than you can in principle monitor if the trajectory – the flightpath – stays inside that region or if it drifts outside it. That is the geometric intuition that is behind our paper. You define a compliance manifold, which a subset of the full state space corresponding to states that satisfy your ontological constraints, and than you build a governance mechanism that redirects the trajectory when it approaches the boundary of that manifold.
Reread that last sentence, and realize you read through it and understand what was written. Now compare that to when you started. Well done, my friend !
That flightpath redirect is only a correction, not a block.
The our OCG, we continuously monitor the model’s flight path through the reasoning space, and when a projected (predicted) move can potentially leave the approved flight-corridor, the OCG ‘autopilot’ applies the minimum adjustment that is required to keep the trajectory within pre-defined (ontology) bounds.
This shifts governance from a hard boundary to active guidance. So this way, we’re not grounding flights no more, but we actively keep the plane on course no matter the wind, the weather and an occasional mirage of a friendly village. This autopilot is what control theorists call a ‘projection operator’. It takes the model’s intended next move in state space, checks it against the constraint boundary, and if the move would push the trajectory outside the valid region, it applies the minimum correction required to bring the move back inside.
The internal world model
My favorite piece of research is a 2022 study by Li and colleagues called Othello-GPT, and I bring it up in every lecture I give on this topic because it perfectly shows just how surprising these systems are and how the surprise turns out to make complete sense once you think about it.
The setup is this. The researchers took a GPT-style transformer and they trained it on sequences of legal moves from the board game Othello. That is all they fed it, sequences of moves, nothing else. And also no explanation of what the pieces meant or what counted as a win. The model’s training objective was simply to predict the next legal move in the sequence. It got very good at this. Then the researchers did something clever, they probed the model’s internal activations to check if it had developed an internal representation of the board state.
Well, it had.
The model had learned to internally track which squares were occupied by which pieces without ever being shown a board. The squares occupation count was implicitly necessary to predict the next legal move. The model reverse-engineered the board from the move sequences because knowing the board state was instrumentally required to do well at the task it was trained on. It did not ‘think’ of the board in a way you would draw it on a piece of paper, but more as the way an experienced Othello player would hold it in their head, I think. More like a functional model to support its prediction, that allows it to think tactics like “if I play here, then the opponent can play there, and then.”
This is what researchers mean by a world model. It’s an internal representation of an environment that allows a system to simulate future states without directly observing them. The model built the board because building one was the most efficient way to achieve the objective.
LLMs that are trained on text appear to do the same thing but at a much larger scale.
The evidence for this comes from probing studies that want to understand if the model’s internal activations encode specific types of structured knowledge. Researchers at DeepMind, Anthropic, MIT, and Stanford have found in different studies, that the internal states of LLMs is all about spatial relationships between objects (causal structures linking events to their consequences) and temporal sequences (time), entity properties and their relations and factual associations between concepts that are causally relevant to outputs. The model does not look up that Paris is the capital of France from a database. It has a compressed internal model of geographic and political structure from which Paris-as-capital is retrievable. That is a world model.
The implication for governance is that, because the model has an internal world model, then the errors it makes are not random.
They are in-fact structured.
And they come from regions of the world model that are incomplete or simply wrong, because the training data was wrong about something or because the query pushed the model into a region of state space where the world model is unreliable.
So, no. Hallucinations are not noise.
But they are the output of a world model operating outside its competent domain.
And that means they have geometry and they cluster in predictable regions of the state space, and those regions can be mapped.
And if those regions can be mapped, they can also be monitored. If they can be monitored, they can be forecasted. And if they can be forecasted, they can be governed..
Got it?
You know know the central idea behind the paper.
We could normally be waiting for a hallucination or a policy violation, etc. to appear in the output, but the OCG now monitors that ‘flight trajectory’ that is going to produce the output. The objective of this monitoring is to determine if the ‘airplane’ is moving toward a ‘village’ – a region – where failure is increasingly likely to happen.
And from this perspective, governance will become more like a prediction problem. Where is the model heading next?
Features, neurons, and the beautiful chaos of polysemanticity
Yeah, I wanted to throw in a big word, so we can all pretend we learned something cool that we can use in our next conversation with an AI vendor.
This is the version of a neural network that everyone learns first, and it is wrong enough to be dangerous. Usually the story goes something like this . . .
Ok, a neural network has – duh – neurons, and each neuron learns to represent some thing. Um, it could be the neuron that fires for cats or for curves or “bank” in the financial sense. If you want to understand what the network knows then theoretically you find the neuron and you read off the concept. That would be darned simple and absolutely interpretable.
Buuuut, you this coming, alas, this is not what happens in large language models, here is where a lot of enterprise AI governance goes wrong.
What actually happens is that neurons in LLMs are polysemantic.
Here we go. Explanation:
A single neuron responds to multiple, apparently unrelated concepts and sometimes a lot of them, depending on context. Anthropic’s 2022 interpretability work was enlighting to me because it showed that individual neurons in small transformers were ‘firing’ / responding to legal terms in one context, to programming concepts in another, and specific syntactic constructions in a third. The same computational unit but with different meanings depending on what else is activated around it. Emphasis on ‘around it’. Now, this is called polysemanticality, and it is apparently the norm of neuronic* behavior rather than an exception.
The reason this happens, by the way, is efficiency, and the mathematical story behind it is important enough that I will explain it in the next section, but for now, the key thing to understand is that the concept of “justice” is not stored in only neuron number 47,382, row 14. It is stored as a specific pattern of activation spread across potentially thousands of neurons, and those same neurons are simultaneously participating in the form of activated regions (just like the brain), in the storage of dozens of other concepts, and all those concepts are overlapping in ways that make individual neurons almost meaningless as a single unit of analysis.
So yeah, you have to treat the inner world of a neural network just like the brain!
Anthropic’s mechanistic interpretability team published a large-scale application of a technique called sparse autoencoders to Claude 3 Sonnet in 2024, and what they found is worth sitting with for a sec. They extracted millions of features from the model – basically directions in the activation space – that correspond more reliably to single concepts than individual neurons do, and they found features corresponding to what you would expect, like cities, scientific concepts, grammatical constructions, historical events. They also found a feature corresponding to the concept of deception. A direction in the model’s internal activation space that, when you artificially push the model in that direction, causes it to behave deceptively. Nobody put it there. It emerged from training on human-generated text, which contains an enormous amount of content about deception, because humans spend a lot of time writing about and around the topic of lying.
Let me rephrase this, for the purpose of better understanding this . . .
We now know that an individual neuron fires for multiple concepts, that is what we call polysemanticity – a neuron has different functions around multiple concepts – and so, Anthropics Pinky and the Brain wanted to know that if concepts don’t reside in neurons, they do appear in patterns across thousands of neurons at the same time. And that is why they used this sparse autoencoder thingy.
Ok, now, take Claude’s activation space (it’s thinking cap) and picture it as one giganormous mixing panel with 20.000 sliders or whatever they’re called. Every concept and association activates a certain pattern of sliders on the panel. That patterns is what we humans find hard to remember, and an SAE breaks those complex patterns down into simpler building blocks. Instead of Neuron 1=0.73, Neuron 2=0.11, it now shows Feature: Weiner dog, Feature: Python code. And they saw that features that correlated strongly with cities or countries, programming languages, DNA, legal concepts and thousands of other concepts, represented a specific direction in the activation space. And that word ‘direction’ is important.
Take a map. You can walk north. That is a direction, which in an LLM can be Paris, Amsterdam, London, Berlin – these words all have the same “city-direction”. Another direction can be “past tence” or another “chemistry”.
So, the model thinks in terms of an enormous landscape of directions, patterns and geometric structures!
Just think of the implications. If concepts coincide with stable geometric structures in the activation space, than this means that compliance, hallucinations, risks and failure modes will also have geometric structures !!
And that is precisely the leap we have made in our paper.
If such features exist, this means that trajectories through this space can be monitored, predicted and ultimately guided. The papers on sparse-autoencoders and residual-stream-geometry were key to this argument because they deliver the empirical evidence that there’s a flight-map inside the LLM, and that we can now build a control tower.
*_just one letter n/t means the difference between a transformer and a therapist.
How to fit a library into a shoebox
The mathematical reason for polysemanticality is superposition, and it was explained rigorously by Elhage and mates in their 2022 Anthropic paper called “Toy Models of Superposition”. I am now going to explain it without the mathematics, because it’s quite intuitive.
Say you have a three-dimensional space. That is three directions, duh, mutually perpendicular, the space you live in. You can have three features that are perfectly orthogonal to each other. Now you need to represent a hundred features in that same three-dimensional space. You cannot do it with perfect orthogonality, but you can do something clever which is you find directions that are almost orthogonal, nearly perpendicular but not quite, and you pack many more of them into the same space. The cost is interference – yes, the same as light – when two almost-orthogonal features are both active simultaneously, they partially activate each other in a way that perfectly orthogonal features would not. The benefit of that is compression because you can represent far more information in a far smaller space.
Example time.
Take X = degree of ‘catness’, Y = degree of ‘paperness’, and Z = ‘degree of carness’.
In that world, life is easy. A pure cat is (1,0,0). A pure sheet of paper is (0,1,0). A pure car is (0,0,1). Activating one feature has no effect on the others because the three directions are perfectly orthogonal. Catness never leaks into paperness, nor does paperness never leak into carness.
Bliss.
The problem appears when you want to represent not three concepts, but a hundred. A three-dimensional space only has three perfectly orthogonal directions. And so, you cannot fit one hundred completely independent features into it.
The trick that neural networks use is to relax the requirement for perfect independence.
They’re not representing concepts using perfectly orthogonal directions, but directions that are almost orthogonal, and these directions are roughly distinct enough to be useful, but they overlap slightly.
In our example, we now add directions for “dogness”, “fur”, “newspaper”, “vehicle”, “city”, “historical event”, and ninety-four other concepts. None of these directions are perfectly independent anymore – because vehicles drive through cities etc – they are packed together into the same limited space. This is when these concepts start to interfere with one another. A feature that represents a cat slightly overlaps one representing fur. Concepts are no longer completely separated.
Anthropic names this phenomenon superposition – just as in waves – and because of this interference, the network can store vastly more information in a small ‘space’ than would be possible with perfectly independent representations.
It trades a little confusion for a massive gain in compression.
Features that are sparse, meaning they are only relevant in a small fraction of inputs, can be packed into the shared space with low interference cost, because the probability of two sparse features needing to be simultaneously active is low. The network is making a statistical bet that if feature A is active only 2% of the time and feature B is active only 3% of the time, the chance of them interfering at the same moment is small enough to accept.
A model with tens of thousands of neurons can represent hundreds of thousands of distinct features, at the cost of a certain amount of ‘cross-talk’ and at the cost of making the features essentially invisible when you look at individual neurons, but the information is real and is still retrievable during inference, because the model has learned both to pack the features in and to unpack them when needed. But it is not organized the way a database is organised. It is more like the way you have all the flavours and techniques to cook a brilliant omelet in your head but not as a labelled shelf system, instead as a web of associations that become coherent only when the cooking actually starts.
Wow.
This compression is genuinely impressive and it is part of why LLMs are so capable relative to their parameter count. It also means that interpretability is going to be very hard. You cannot read off what the model knows by looking at individual neurons. You need techniques that decompose the distributed, superposed representations back into something human-readable.
And this is exactly what sparse autoencoders are designed to do, and it remains an active research problem, and also a cool thing to regurgitate at your next meeting about explainability.
Why the model gets stuck in bad neighborhoods
Back to the airplane metaphor, which I am going to develop a bit further now that you have the state space concept from section two.
The model’s internal state during inference is moving through a high-dimensional space. It is not moving randomly but being pulled toward regions – by the geometry – that the model’s weights have learned are high-probability, stable configurations given the input. These stable regions are called attractors. These are the villages the airplane tends to land in.
But the concept of an “’energy landscape’ is another way to describe the same thing.
Instead of the geometric approach we used earlier, with peaks and troughs, the state space is now flattened into something you could walk across. Kind of like my country the Netherlands. The low-energy regions, the valleys, are the attractors, the places the system tends to settle into and the high-energy regions, the hills and ridges in our previous model, are now unstable configurations the system passes through quickly on its way somewhere more stable. The shape of this energetic attractor landscape is determined entirely by the model’s weights, which were shaped by the training process, which was shaped by the training data, which was shaped by the collective of humanity.
In practice, this means that different models have different landscapes. A model trained on high-quality, diverse, carefully curated data has deep, well-separated valleys in the right places. It settles into accurate states because those states are genuinely stable in its landscape. A model that is, say, trained on lower-quality data has a much messier landscape with shallower valleys or they may be in wrong places or ridges in places that should be open plains. The model will still settle into valleys, but they are not where you want them.
This landscape analogy also explains something about prompt engineering that we as practitioners understand intuitively but we rarely articulate in a formal way.
A prompt is a specification of a starting position in the state space. Where you start matters because the landscape around the starting position determines which valleys are reachable. A good prompt places the starting state in a region of the space where the nearby attractors are the ones that you want and doesn’t start where the nearby attractors are things like hallucinated citations, or plausible-sounding nonsense.
The Attention-Deficit-Loop framework (’Attention isn’t all you need’, paper in comments) we published in 2025* is partly about exploiting this geometry for creative generation. Standard LLM inference follows high-probability paths through the landscape where it settles into the nearest deep valley. In the Wander phase of the AD-Loop I deliberately let the model take larger steps in the embedding space, and have it explore shallower valleys and hillsides that standard inference would skip past. It also takes samples from broader neighborhoods of the current state. The goal of this exercise is to discover features and combinations in low-probability but high-creativity regions of the landscape that standard generation would never reach. It is, to extend the metaphor I’m using, the difference between taking the main road to the nearest town and wandering off the marked path to find the village nobody visits that turns out to have the best restaurant.
*_Read Attention isn’t all you need: The wanderer’s algorithm | LinkedIn
Hallucinations are simply physics
This section is the one I feel most strongly about, because we tend to frame hallucinations as a ‘bug that will be fixed in the next version’, and that has cost enterprises an enormous amount of money in buying or building naive deployment architectures, and I would like to do my small part to displace it.
A hallucination happens when the model’s trajectory in state space settles into a stable (but false) attractor that corresponds to a plausible but factually incorrect configuration. The output appears as confident because the attractor is stable. It is also fluent in text and grammar because stable attractors in a model that is trained on human text are configurations that produce grammatical and stylistically coherent language. But in the end, the output is wrong because the attractor is in the wrong place in the landscape. And that is either because the training data contained misinformation, or the query pushed the state into a region where the relevant attractor is a false one, or maybe the model is simply being asked about something its world model does not cover very well.
A cool paper from Cherukuri and buddies somewhere this year, formalized this in what they call ‘hallucination basins’. Those are identifiable regions in the model’s latent state space that are corresponding to stable but incorrect configurations. The geometry of these basins is shaped by the same training process that shaped the rest of the landscape, and it is sculpted by the statistical structure of errors in the training data and common human misconceptions and other freaks of nature like co-occurrences of concepts that happen to be wrong. The inherent tendency of the model is then to extrapolate pattern-completing responses into domains where the patterns do not hold.
Visualization time . . .
The easiest way to think about a hallucination basin is to think of the model’s internal state as a landscape with valleys. You ask the model a question, and that question begins moving through the landscape and when it is further down the line of processing your prompt and is generating tokens, its internal state follows a trajectory and eventually that trajectory settles into a stable region. Kinda like a marble rolling downhill and resting in a valley or electrons in their ‘orbits’.
Most of what you ask the model will end up in a useful valley. “What is the capital of France” will take you through a trajectory through regions that are associated with France, and capitals and eventually converges on Paris. But some of these valleys are just plausibly sounding nonsense, and your trajectory has entered a stable region of the landscape that consistently produces an incorrect answer – with the emphasis on ‘consistently’.
Now, why do these basins even exist.
The model learned its landscape from us, the data, which contains these patterns and human biases and incomplete or simply false stuff.
A paper from Akarlar in 2026 added something that should make anyone who is running high-stakes deployments uncomfortable. Hallucinations are often committed before the first word of the output appears.
Yes. Prompt → Trajectory → False attractor → Land → Boom! Wrong.
Akarlar used a technique called same-prompt bifurcation*, where they ran the same prompt a few times and observe where the trajectories diverge. By the way, did not literally watch a little dot flying through latent space, but they ran a prompt multiple times and since the model samples tokens probabilistically, some runs produce the correct answer and some produce a hallucination, and for every layer of the network and every token position, they recorded the internal activations by just saving the resulting vector (which is a big array of thousands of dimensions wide, like [0.13, -2.44, 0.88,→], whatever, you either get the picture, or you don’t care ;).
The researchers saw that factual and hallucinated outputs diverge at or before the first generated token. The model “decides” to hallucinate, if you will forgive the intentional verb, before it has written anything and when the trajectory has entered a hallucination basin, if you want to correct it, that requires a sustained intervention across multiple layers of the model. They also injected a hallucinated activation into a correct trajectory, and it turned out to corrupt it easily, but redirecting a hallucinated trajectory back to a correct one is quite harder.
And this is why governance must be pre-inference rather than at runtime or post-hoc.
By the time the model has produced a hallucinated output, the damage is done, and reversing it is very hard if not impossible because the state has settled into a false attractor and the first tokens are already wrong, and the cost of detection and correction is higher than the cost of prevention would have been. The OCG is designed to catch trajectory before it even starts and our second paper catches the deviation early, during inference, while the state is still in motion, before it has locked into a basin we do not want it to be in.
*_The word bifurcation comes from dynamic systems theory and is about a point where one trajectory splits into two different futures. Yes, they simply could have said that, but hey, that’s why they choose to become academics.
False attractors and the geometry of getting it wrong
Congratulations, my smart friend! You have made it this far and you haven’t given up! This means you are a person I’d like to connect with. I like to work with smart and eager beavers, so drop me a DM on LinkedIn. There’s only two people on this planet with my name, the other one is related, so he’ll revert you back to me if you end up with this ‘false’ attractor.
Talking about which . . .
As said earlier, a false attractor is a stable region of the model’s state space that corresponds to a pattern that is plausible but wrong. The model is not trying to lie to you or anything because it doesn’t have any intentions, but it is simply settling into a stable state the way water or a marble settles into a low point, but unfortunately, the stable state happens to be factually incorrect.
Having said that, the most dangerous false attractors are what I call confabulations. These aren’t factual errors like “the capital of Australia is Sydney” (it is Canberra, a fact that LLMs have gotten consistently wrong for years because Sydney is more famous and appears far more often in training data), but they’re fully fabricated and internally consistent but they have no referent in reality. Take a confabulated biography for a real person. Like the Harvard professor who got accused of sexual harassment of a student by ChatGPT*, I wrote a short post about it back then, but don’t want to refer you to it, because the man got enough false flak as it is. We have seen many confabulated cases in real court proceedings recently, all with docket numbers and dates.
The reason confabulations are more dangerous than simple factual errors is that they pass surface-level plausibility checks. The sentence structure is always correct with appropriate vocabulary and the level of detail matches what real information in this domain would look like. There is no surface signal that the underlying content is invented.
Mode collapse is a related phenomenon.
It was detected originally in generative adversarial networks that have existed since 2014, in which the generator learned to produce a narrow range of outputs that fooled the discriminator which never explored anything else. Mode collapse in LLMs can be described as ‘geometric collapse**’. The trajectory loses access to the full state space and becomes constrained to a low-dimensional region from which it cannot escape. The model then starts to produce repetitively confident outputs in a narrow band of responses, because its internal state is stuck in a constrained region of the manifold.
Explanation time . . .
The model is temporarily suffering a dramatic drop in, say, it’s IQ – effective intelligence. The knowledge is still available to it and the parameters have not changed, and it hasn’t forgotten a thing, but its trajectory is trapped in a small but significant region of the state space, and so it is cutting it off from much of the reasoning landscape that you would normally be able to access. Yeah, it’s IQ suddenly dropped from genius to moron with a dash of Dunning Kruger because it becomes repetitive, overconfident, unimaginative, and unable to escape a narrow range of responses, even though it still has greater capabilities in principle.
Now, what is interesting to our unsexy frontier of AI, is that this happens more often in long-horizon agentic tasks, where the trajectory has accumulated many steps and the compounding of small constraints has pushed the state into a corner of the space.
So yes, this matters to enterprise deployments, and this is specifically the failure mode I see most often in multi-step agents where the agent runs twenty steps of a procurement workflow and somewhere around step fifteen the trajectory has drifted into a region of the space where the available responses are all variations on the same wrong approach, and the agent produces step sixteen with full confidence that it is following the correct procedure, and step seventeen, and step eighteen. And all of that happens without anyone noticing until the invoice lands on the wrong customer’s desk.
The AI labs have spent considerable time and effort in handling this problem, but none of them try to change the geometry of the model itself, because that is hard and expensive, and it requires retraining. Most techniques try to give the trajectory better landmarks through the use of RAG that injects external facts into the context window so the model has something reliable to anchor itself to, and reflection loops ask the model to critique its own work, and with multi-agent systems, you introduce additional perspectives that challenge a developing error before it becomes locked in.
How does that work?
Say you have an agent that thinks “supplier A has a contract X”, but that contract does not exist. But the agent doesn’t know that it’s hallucinating, so it continues further, but based on wrong assumptions, contract X has clause Y, so invoice Z can be paid out etc. Every following step seems logical, but the error is deep inside the model. Now we add a second (adversarial) agent to the mix with the same task, but a different assignment – it is always looking for evidence against a claim made by the first agent. It always challenges the agent. Bit like that obnoxious colleague who always second guesses you in meetings. But the thing with this approach is that the second agent could also be wrong, because it is trained on the same data or uses the exact same model. In one of my earlier papers, I disproved this method (link in comments) as trustworthy for enterprise use, simply because when two agents share the same blind spot, they can reinforce the same mistake. Five agents can agree on something that is completely wrong, and so consensus cannot pass as evidence.
So the industry falls back to things like verification pipelines, and fact checkers, knowledge graphs, and guardrails that attempt to detect when the model is drifting away from reality and provide corrective signals before the drift is significant. In our airplane example, they are nothing more then navigation aids. But the reality is that all of these techniques operate after the trajectory has already begun to drift. They improve the odds, sometimes dramatically, but they rarely address the underlying dynamics that produced the failure in the first place.
This is why so much current research is shifting toward trajectory-level monitoring and activation steering and geometric approaches to governance. The goal of these measures is to understand and steer at runtime, where the model is heading while it is still thinking. If false attractors, hallucination basins, and geometric collapse are fundamentally properties of the trajectory then the most effective intervention point is somewhere along the flight path before the model settles into the wrong valley.
And that is why, I know that the future of enterprise AI lies in the adoption of ontologies. And yes, this project is going to be a bitch, but sometimes you need something nasty when you want good results.
*_When you type in the name of this man + sexual harassment, you will see that ChatGPT goes dark, and you have to start a new session. They simply hard coded the problem away because they couldn’t get the false attractor / confabulation out of the system
*_Paper: ”Escaping Mode Collapse in LLM Generation via Geometric Regulation”
The accidental neuroanatomy of something that was never supposed to have one
I have a habit of presenting this finding at the end of lectures and watching the room go quiet when I show them a gradient descent diagram.
Large language models who are trained exclusively on text with no reference to biology or neuroscience, appear to develop functional analogues to brain regions.
Yes.
Specific subsets of their artificial neurons become selectively responsive to specific types of information ijust like the functional specialization of regions in the human brain, and this specialization was not designed in. It emerged.
In 2025, AlKhamissi and his Oompa Loompas used a neuroscientific approach to identify causally task-relevant units in LLMs – the parts of the model that are actually responsible for a particular capability – and he found what they called ‘language-selective units’, subsets of the model’s artificial neurons that respond preferentially to language tasks, and when he removed them, it caused a disproportional degradation in language performance with little effect on other tasks. Yes, they found specific neurons that act like language specialists, and the model became much worse at language when they were removed, but all the rest remained the same. These units cluster in the model’s architecture in patterns that quite resemble the organization of the human language network like the cortical areas and Broca’s area and Wernicke’s area that biological brains use for language comprehension and production.
Cool huh! Without having to teach it biology.
Sun and his mates, somewhere in 2024, set out to extended this research and they mapped subgroups of artificial neurons in LLMs to specific human functional brain networks, including something called the “default mode network’ – the brain regions that are active when we’re not focused on the outside world and when we’re engaged in internal thought – and also our visual processing system, and our language network. They saw that this brain-like functional organization is common in more advanced models. So, the more a model is advanced, the more it starts looking and behaving like a brain including regional activity centers. The gap in brain-like organization between an early BERT model and the recent Llama family is measurable and quite large. Something that happens as these models scale, some dynamic in the training process at sufficient scale, something appears to push the model’s internal organization toward patterns that look very much like the way biological intelligence is organized.
And the thing is that no human planned for this to happen. They were not building models to resemble brain regions – ok, ok, Rosenblatt’s Perceptron was based on a simple neural network, and we still call the nodes in a neural network, a neuron. But that’s it, we are minimizing a loss function on text prediction, and the minimization process is converging on structures that look like brains, presumably because brain-like functional organization is apparently an efficient way to process the kind of information that human language encodes, and the optimization process finds that efficiency whether or not we asked it to.
Brilliant, but strange nontheless.
But there is a caveat that I want to be clear about because there’s this temptation to over explain this finding. Resemblance does not mean equivalence. LLMs process all input tokens in parallel (that’s why an LLM likes GPUs), and then they compute attention across the full context window. The human brain processes language recurrently, that is word for word, with constant feedback between earlier and later representations. The training environment for an LLM is text and the training environment for a human brain is embodied, social and temporally extended existence with touching things, falling down, watching your parents’ faces, learning what anger sounds like and screwing things up. In other words, this does not mean the model has a self, only that certain organizational principles for processing language-structured information are quite universal across different substrates. That’s all.
What it means practically is that the internal organization of these models is not random, and probing that organization is a meaningful activity that tells you something real about what the model can and cannot do reliably.
Your AI forgot what you said three messages ago
Alan Baddeley published his model of the human working memory in 1974, and it remains the dominant framework to this day. The version I use in lectures has four components, which I will describe quickly because the contrast with LLMs is the point.
Baddeley’s model has a ‘central executive’ that takes on the role of the attention director. It decides what we focus on and it coordinates the other components. The phonological loop – the component of memory that temporarily stores spoken information – is the inner voice that holds verbal information alive through internal repetition. This is the mechanism that lets you keep a phone number in your head by repeating it until you write it down. The visuospatial sketchpad is the inner eye that holds visual and spatial information, the mental map you use to navigate a building you have been in before. The episodic buffer is the component of the executive function that integrates information across all these components and connects them to long-term memory, and that apparently creates a coherent experience for us all.
The critical property of human working memory is that it actively manipulates information internally. You can hold a number in your head, rotate a mental image, run a rehearsal of a conversation you have not had yet, that sort of thing. The information does not need to be externally present, you simply can held and process it internally, without being written down or spoken aloud.
LLMs don’t have this. What they have is a context window which is the segment of input text that is physically present in the model’s input at the time of inference. Everything inside that context window is accessible to this component called the attention mechanism, and all that is outside of it does not exist for the model. This means that the model has no internal state that persists between inference calls, and so, each call is a fresh instantiation.
The context window is the entirety of accessible history.
Huang etc. wrote a paper in 2025 called “LLMs Do Not Have Human-Like Working Memory”. He developed a Number Guessing Game task that required the model to maintain an internal number across a conversational sequence without externalizing it (storing it somewhere), the way a human would hold a number in mind. LLMs failed this systematically. They cannot hold information internally without externalizing it. Once information leaves the context window, it is gone. Poof.
This is why chain-of-thought prompting works, by forcing the model to write out its reasoning steps, you are creating the external written trace that compensates for the absence of internal working memory.
The model cannot hold intermediate results in mind, so with this technique, you’re giving it a notebook and making it write things down, and the written things then re-enter the context window as additional input, and then the model can use them. It is a clever trick.
The implication for the enterprise is that any system architecture that is relying on the model “remembering” something from earlier in a conversation without that something being present in the current context window is architecturally broken.
Each inference call is the first moment of its existence, and the context window is everything it has ever known.
Everything the model knows is baked into its bones at birth
If the context window is the working memory, then the weights are the long-term memory.
Funny huh !
Everything that a large language model knows about the world is encoded in the numerical values of its billions of parameters. Every fact or association, pattern or relationship, everything, is all in the weights*. And the weights are frozen – ok, you can get half-baked models like LoRa, but you get the idea – they do not change during inference. They were fixed at birth – at the end of training – and they stay fixed until someone retrains or fine-tunes the model. The model’s long-term memory is a static snapshot of its training data and it’s compressed into weights and accessible through inference but not updatable from inference.
This has several consequences that seem obvious once you state them but that organizations routinely ignore. The model does not learn from your conversation. It does not update when you feed it your business plan. Telling the model “that is wrong, the answer is X” does not change any of its weight. It adds a correction to the context window, that the model can use to generate a different output in the current conversation, but the next conversation starts from the same frozen weights with the same incorrect association intact.
And no, the way the weights encode factual associations isn’t like a database. You cannot open the weights and find the entry that says “capital of France equals Paris.” The association is distributed across the weight matrices of the feed-forward layers and attention mechanisms, and it’s encoded in the geometry of the parameter space but it’s not readable by humans. This is both what makes the model so powerful (enormous amounts of structured knowledge can be compressed in a few billion weights and neurons) and it’s at the same time also that what makes interpretability so hard (the knowledge is smeared across the entire network in overlapping, superposed patterns).
The 2022 ROME paper by Meng and friends made a significant contribution in this domain. The researchers recognized that specific factual associations in GPT models are localized to identifiable computations in the middle-layer feed-forward modules and not distributed across the entire network equally. Cool innit. But instead they’re concentrated in specific computational steps that can be targeted. They showed that you can locate the weight patterns that are responsible for encoding a specific factual association and edit them directly, and that changes what the model believes about a fact without you having to retrain the entire network!
If the model incorrectly believes the Eiffel Tower is in Berlin, you can find the computational locus of that belief and correct it surgically.
This is the closest thing we can do in current AI-tools-arsenal to targeted memory editing.
And it is an important capability for enterprise deployments where the model’s knowledge is outdated or domain-specifically wrong. It also tells us that long-term memory in LLMs is not uniformly distributed across all weights equally. It has internal structure, internal geography, and that geography can be studied and, to some extent, navigated.
This a domain where I haven’t seen any products, so this may be a commercially interesting proposition for someone to develop. Say a company changes a policy that you need for your agents (approval limit goes from 10k to 25k), today we solve this with RAG or prompts, but a tool that could continuously probe the model to find where facts like approval limit lives internally, could create a knowledge dashboard. You could even show that Policy A = aligned, and Policy B = partially aligned. That sort of stuff. You could create a hotfix for corporate memory. And if we take it even further, you could create a hallucination hotspot detection as well.
Ok. I’ll wait.
*_It just dawned on me that Weight Watchers is actually the physical representation of quantization. Hahahaha. . . ok, if you can laugh about this, drop me a DM, you belong at Eigenvector!
The thing that should terrify you about all of this
We do not know the extent to which the internal world models of production LLMs are accurate.
The Othello-GPT case was verifiable because Othello has closed rules and a definable truth function. But the real world does not. We can probe internal representations and find correlates of factual accuracy, but we cannot run the equivalent check across the full scope of what a production model is asked to do.
We do not know how many deception-adjacent features, overconfidence features, or plausibility-over-accuracy features exist in production models, how often they are activated during inference in typical enterprise use cases, or if the monitoring architectures we have would catch an activation of those features across our deployed workflow.
We don’t know either how trajectory validity degrades in genuinely long-horizon agentic settings, where the model’s state has been influenced by hundreds of tool call results, retrieved documents, intermediate reasoning chains, and conversational turns, where each of which has pushed the trajectory in some direction. In our OCG paper we prove the insufficiency of action governance mathematically. The empirical characterization of how trajectory drift accumulates over long horizons in production systems is still an open research question.
And we still don’t know if the internal representations that interpretability research identifies are the actual computational causes of the model’s outputs, or if they are correlated with those causes but not identical to them. Mechanistic interpretability just assumes that a feature that correlates with a concept and causally influences output when perturbed is telling you something true about the model’s computational structure, and this assumption may probably be right, but it’s not proven.
And none of this stops us from deploying agents in our company.
But then again, nothing us humans build is perfectly understood at the level of mechanism. Take my car, it’s not perfectly understood, and still it gets the job done The financial instruments that destroyed the 2008 economy were not perfectly understood either by the people pricing them, and those people even had PhDs in mathematics.
The inner world of an LLM is more structured and geometrically coherent than most enterprise AI practitioners give it credit for. It is also bloody opaque and more failure-prone in specific and predictable ways, and even more dependent on careful architectural management than most vendors even know about their own products. That is the state of the field in June 2026.
My Weiner Slob has no explicit model of how his olfactory cortex constructs a coherent representation of the world from molecular signals. He simply sniffs effectively anyway, to the detriment of the neighborhood’s cats. The question is if navigating effectively is good enough when the navigator is making procurement decisions and writing compliance reports.
I remain convinced the answer is no, and I remain equally convinced that I will deploy six more systems this quarter. The tension between those two positions is where I live, and I suspect it is where most of you live too.
Signing off,
Marco
Ding dong! Commercial.
For my daytime job, I’m a researcher and factory builder at Eigenvector, a commercial research lab operating at the frontier of unsexy AI. Eigenvector explores how autonomous systems, sovereign AI, agentic governance, and enterprise architectures can be transformed into production-ready capabilities. With our research programs, we develop new frameworks, models, methodologies, and operating concepts for Enterprise AI and with our AI factories, those ideas are validated and industrialized in real-world organizations.
NCC-1701 
Leave a comment