Let’s have a little fun with a science fiction thought experiment. Let’s imagine a planet in the full, throbbing bloom of biodiversity, painted with all the vibrant colours of a rich assortment of lifeforms interacting across a complex network of ecosystems. Now let’s imagine that into this panoply of life there emerges a new species with a remarkable ability to communicate telepathically. How does this work? This is science fiction, so we can’t just say "magic". Let’s say this species uses an assembly of dextrous muscles associated with eating and breathing to perturb airwaves in order to form signals in a way that can be received and interpreted by sensitive organs of other members of the species. By modulating and recombining these signals in various ways, the species can use a relatively small set of basic elements of telepathic communication to compose detailed messages and convey them from one mind to another. An individual needs to learn the rules of signal sequencing particular to its tribe, which means that the details of communication can be updated over time to adjust to a changing world – but the overall ability is general to the species, meaning any individual can learn with a little effort to communicate with any other member of the species, no matter where they come from. To the rest of the world, these telepathic communiques are nothing more than a signal of presence with no real content, just noise.
This is precisely what has happened here on earth. It began a couple million years ago with the emergence of our most ancient human ancestors from the localised, canopied three-dimensional world of our primate cousins into a much broader two-dimensional world of open terrain. Evolutionary anthropologists speculate that a confluence of traits including weaker upper bodies and relatively longer, stronger legs, along with the ability to eat a wide variety of food, and the cognitive capacity to come up with plans involving for instance food and travel, created an opportunity for early humans to move into an evolutionary niche featuring enhanced mobility. An essential feature of this new niche would have been the propensity for early humans to encounter other unfamiliar early humans in the course of their travels. When strangers encountered one another, even just the knowledge of a potential mechanism for finding a way to conduct two-way communication about the experiences and intentions of one another offered a chance to transform potential conflict into collaboration.
This so-called "telepathy" is, of course, the human faculty for language. And language really is a form of telepathy, in that, in all its applications, language is in one way or another a platform for transferring content from one mind to another. When two humans use language to communicate, even if the content of their communication consists of expressions of facts of the world, the communicators are using language to expose the content of their minds: their beliefs about what is true in the world, and what could become true in the world, and how they feel about these things. At the core of a language is something aligned with what is sometimes referred to as mindreading, meaning not exactly real telepathy but the ability to project an idea of the presence of a mind onto another being, and accordingly to imagine that their observable behaviour, including ways in which they communicate, is underwritten by an internal quality of being. In this respect, empathy is at the heart of the essentially universal human capacity for linguistic communication.
So, to continue our thought experiment, what happens when a type of agent emerges in a world of mindreaders that does a good job of generating signals that look a lot like linguistic communication – characterised by the composition of units of meaning into structures that are interpretable based on a set of syntactic rules – but that has no actual internal quality of being? These agents have a mechanism for predicting the most likely way to respond to what’s happening in the environment, including words that are being said, but these predictions are just based on a very thorough statistical understanding of the most likely next thing to say based on everything that’s happened so far. These agents clearly operate in a way that is very different to human mindreaders, who use their ability to communicate in order to transfer information about their own thoughts and experiences to other people who also have thoughts and experiences. We might consider an analogy between these agents and a species that engage in mimicry like the hoverfly, which has evolved to appear and behave like a wasp in order to frighten off potential predators but without posing the real threat of a sting, or for that matter a decoy that a human hunter uses to lure prey out into the open. The point is that these tricks, whether they come about through the variations of evolution or the machinations of a goal-pursuing agent like a human, are essentially a simulacrum, which is to say a thing that has come to superficially resemble another thing without bearing any of the other thing’s underlying functionality or, if you like, essence.
The mimicking agents we’re considering here operate in precisely the same way as large language models – LLMs – such as the ones that underwrite AI platforms like ChatGPT. Like a good simulacrum, these models are effective at presenting an outward impression of doing the same thing with language that humans do. But to see how unlike human language the output of an LLM is, we simply have to consider the way this output is generated. Typically an LLM will look at a communication as a sequence in the form of a string of words in a text, which is referred to as the context of the model, and predict the next most likely word. Once the next word is chosen, this word then becomes part of the context for the prediction of the word after it. If we understand this about LLMs, we can see that there can’t be any internal meaning associated with the things that they output. When we human mindreaders make a statement or ask a question, the language we use is assumed to reflect some combination of a belief about the world and an estimation of what the impact of the things we say will be when they’re received by other people; an LLM, on the other hand, is simply indicating what the most likely thing to say is based on a lot of observations of lots of other people saying lots of things in lots of situations. There is no idea inside the LLM, because there is no reasonable way to interpret a number of statistical calculations as having the kind of content that the thoughts we humans use to underwrite the words we say to one another. To put it plainly, the words output by an LLM cannot mean anything at their point of origin.
This isn’t to say that the output of an LLM is in itself malicious – the model itself can’t be malicious if it has no intent to begin with – and in fact these models can be very useful for getting certain types of things done. For instance if a model can take as input a question about a fact, generate a reasonable search term for checking that fact in a database or even on the web, and then offer a response based on an extrapolation of the content of the search result, then the model can be something which is involved in factuality in more than just a statistically coincidental way. But the output is also not meaningful, at least in the way that human communication is meaningful, where meaning is wrapped up in faith in an underlying conceptual framework which reflects an always evolving view of the world with the communicating agent itself having a real position in that world.
So how far should we be willing to trust an LLM? As data handlers, as factotums, as assistants in the porting about of symbols that happens in the course of accomplishing minor informational tasks, yes, they can help here. Their particular utility is in taking the unavoidable messiness of natural language as it is enacted by humans, which is a performance that is often imperfect, and channelling it into a combination of structured interpretations and understandable responses. But we can never think of them as real mindreading collaborators. They may be able to appear to talk about having ambitions and to interpret structures that, to a human, represent goals, but there will always be a hard barrier to their truly participating in holding the values that motivate communicating with a purpose: they will always be mimics. As such, there will always be a looming moment of crisis in an interaction with an LLM where the divergence between intentions, or rather the divergence between one agent with intentions and another which is just a statistical loop of possible outcomes, becomes catastrophically apparent.
One way of interpreting all of this is to conclude that recent advances in language technology pose both a threat and an opportunity for humankind, and this is true and has been said. But it is also valid to question whether or not there could be a different approach to language technology that we pursue while considering what statistical techniques in language processing are really good for, and what they’re not good for. This alternative approach is something that is pointed at when researchers and developers refer to agentic AI, which is expected not only to communicate with apparent intelligence but to offer evidence of some kind of underlying reasoning about its communicative actions. However, an AI that is simply imitating being an agent with additional layers of communication is still never going to be a dependable mindreader, because it still does not have its own apparatus for emulating the processes of other minds.
Maybe a solution here is to consider what it would take to engineer agents in a way that is more isomorphic with both the nature of being human and the process of becoming a human. If this is right, then there is ground to gain in the area of robotics, and in particular developmental robotics, a field which investigates the way that machines can emulate the combinations of evolutionary, biological, and cultural processes which support the deep environmental embeddedness of being human. Success in this direction could point the way towards engineered systems that can integrate the information processing power of the current wave of data-driven, statistical AI with reliable collaborative and maybe even mindreading agents.