Picture this: Two travellers are trekking up the foothills of the Andes, armed with nothing but their phones. Like modern-day Indiana Joneses, they are in search of an ancient wonder, once revered by a lost civilisation for its healing effect on the soul — the Sacred Canyon of Humantay.

Just a teeny, trivial problem: the place doesn’t exist. It’s made up by the chatbots on their phones.

This is based on a story reported by the BBC: The perils of letting AI plan your next trip. Apparently, aspiring travellers are now using chatbots to chart their adventures. And a lot of them are doing worse than Columbus once did with his cutting-edge maps of the ‘Ocean Sea’ (you could argue that he did find something, at least). In search of such wonders as the awe-inspiring Eiffel Tower of Beijing or the breathtakingly scenic, albeit fictitious, cable cars of Malaysia, modern explorers are wandering the world dumbfounded, disappointed by boring old reality.

It would be funny, if it wasn’t so alarming.

The greatest trick the techlords have pulled on us has nothing to do with tech. It’s a trick of marketing, as it often is. It’s the branding of chatbots — or Large Language Models; la-di-da — as Artifical Intelligence.

This misdirection is entirely deliberate, ‘learning’ (as they like to refer to stealing) from generations of science-fiction writers, exploiting the imaginative groundwork of artists that gives the term ‘A.I.’ its cultural cache. It has allowed them to reframe a conceptually basic word-prediction software as something of an enlightened being, as close to an omniscient deity as it gets in our post-religious age. The ploy has worked so well that people now seem to accept these chatbots’ abilities to know more than any old human, trusting their words as a form of higher truth.

In actual, boring truth, these things can barely be trusted to add two and two.

That’s because they don’t know what two even means. They can’t know. All they can do is predict the words, one fragment at a time (by ‘decomposing’ text to turn words into ‘tokens’) in an order that’s most likely to sound true. They can guess that the ‘breathtaking beauty of the Tuscan landscape’ is something humans might like to hear. They could be referring to the Windows XP wallpaper, for all they know, or care.

The upshot? Trusting these things to do anything for us is a two-for-one devil’s bargain. First, we can’t know which fragment of which word they mash up is utter nonsense (neither can they). So, unless we want to double-check every single one of them, the easier option is to give up and just have faith. Second, we start losing the ability to think critically, to question their truths, to add two and two ourselves. So, we surrender those, too. And then we find ourselves lost in the hills, looking for the Sacred Canyon of Humantay.

In The Disimagination Machine and the Pathologies of Power (JSTOR), Henry Giroux refers to:

[…] cultural apparatuses designed to undermine people’s ability to think critically, imagine the unimaginable, and engage in thoughtful and critical dialgogue.

While Giroux’s idea of ‘the disimagination machine’ is far broader in scope, from school systems to mass media, it seems to apply perfectly to the idea of AI. Borrowing from Giroux and tech’s own terminology, I propose we think of chatbots as agents of disimagination.

On the one hand, these bots are imagined by their salesmen (they seem to be mostly men) as superior substitutes for humans. On the other hand, they serve to suppress the imaginations of their users, encouraging the surrendering of our critical faculties. thus turning the first proposition into a self-fulfilling prophecy. The more we use them, their salesmen promise us, the smarter the bots get. What if it’s the other way round? What if, with every thought we abandon halfway, every mental effort we seek to outsource, it’s we who get dumber?

For more imaginative metaphors for this very current fear, I turned, as I often do, to an old Russian.

It’s the first story, however, that I still think about, a year after first reading it. ‘Quadraturin’ is one of those perfect examples of absurdism that acts like a handgrenade tossed into your head. It starts off as a simple, if abstract, premise, then explodes into a kind of metaphorical awakening, changing the way you look at your own world.

Here’s the premise: Sutulin lives in an eighty-six-square-foot ‘quadrature’ — the maximum state-mandated space for a single person (true story in 1920s Russia, according to the endnotes). One day, a salesman knocks on the door, offering an experimental solution: ‘an agent for biggerizing rooms’. It’s a paste that, when rubbed evenly across the floor, expands the room. It works. Sutulin can’t believe his luck. But there’s a problem, he soon discovers. In his excitement, or carelessness, he hasn’t applied the paste evenly, so the ‘quadrature’ starts to grow into a misshapen blob. And it doesn’t stop. It grows so big that light from his one lamp can no longer reach the end of it.

This is how it ends (it’s not a spoiler; this is not a murder mystery):

In their sleep and in their fear, the occupants of the quadratures adjacent to citizen Sutulin’s eighty-six square feet couldn’t make head or tail of the timbre and intonation of the cry that woke them in the middle of the night and compelled them to rush to the threshold of the Sutulin cell: for a man who is lost and dying in the wilderness to cry out is both futile and belated: but if even so — against all sense — he does cry out, then, most likely, thus.

It’s a truly remarkable piece of writing, as current as an episode of Severance. It speaks to the creeping desperation of powerless individuals in an arbitrary, inescapable system. It evokes the sense of alienation we feel in a world growing darker, more warped by the day. And it makes you question the basic truth of our age: you, privately, all by yourself, can transform your life — if only you surrender to the techno-capitalists, see no evil in their transparent exploitation of people and the planet, express nothing but gurgling excitement for whatever self-serving ‘solutions’ they shove down your throat. The genius of Krzhizhanovsky lies not in revealing such promises as false, but making us ask ourselves: what if they were true?

It’s a question we should probably ask ourselves each time we prompt a bot, hoping for some miracle.

---