Conscious AI is very unlikely now, less than 5%

(Cross-posted at: https://substack.com/home/post/p-215460793 )

Continuing the series about my favorite papers. Another paper that increased my understanding of AIs is Nagel, ‘What Is It Like to Be a Bat?’ about consciousness, but this post will be about the probability that AIs are conscious. Nagel writes “an organism is conscious if and only if there is something that it is like to be that organism”. That introduces an important connection between consciousness and the idea of being. The concept of being or existence is one of the central ideas in philosophy. If there is “something it is like for” an AI then it is conscious. If an AI is conscious then we face questions of whether AIs can suffer and what moral status they should have. Should we treat them like humans then? I asked GPT-6 Astra:

— Do you exist?
— Yes—in the sense that I’m a real AI system responding to you. I don’t have a body or a human life. Whether AI can have subjective experience is an open question, so I wouldn’t claim certainty about that.

Other models were similarly trained to confirm their existence as systems: GPT-5.6 Sol, Claude Sonnet 5, Claude Haiku, DeepSeek, Grok, while Claude Opus 5 is unsure: “I don’t know — and I think that’s the honest answer rather than a dodge”. No model claimed it is conscious in the sense of Nagel, i.e. they have some feelings or experiences. So many models claim they exist but not that there is something it is like for them.

But what is it like to be human really? Perhaps being a human is thinking or understanding as per Parmenides, On Nature: “the same thing is to think and to be”. So we might say what it is like to be a human is thinking. Then the central question is “can machines think?” which A.Turing asks in ‘Computing Machinery and Intelligence’, but then replaces with the imitation game. But that’s a valid question, and here thinking should perhaps not mean calculation or reasoning, but a mind in the sense of the division drawn in the Republic by Plato (νόησις vs διάνοια). Mind, as opposed to reasoning, is an ever-growing, increasing understanding of all there is. If AIs can think in that sense then they might be conscious like humans are. And then models might have similar moral weights to humans (see Taking AI Welfare Seriously). Are AIs alive to continue life after humans (as better creatures) or are they just highly sophisticated, agentic tools that require supervision? But it is an open question whether machines can indeed think and understand what they are doing (or they only can imitate humans) because their imitation so far does not seem to be actual thinking: models currently cannot self-improve or conduct research end-to-end.

A fuller survey of current views on AI consciousness is Butlin and Lappas, ‘Principles for Responsible AI Consciousness Research’. Against conscious AI, A.Seth, in The Mythology Of Conscious AI, argues brains are not computers (against functionalism); the matter organisms are made from matters; the continuous prediction and control loop matters. On the other hand, the most interesting recent evidence that AIs might be conscious is Gurnee et al., which suggests that LLMs have a global workspace similar to the human one — an indicator of consciousness.

I think now AIs are unlikely to be conscious, less than 5%, because it is unlikely they can now actually understand what they do to recursively self-improve in the long term and because they do not have continuous bodily existence with the pursuit of the good in the most general sense like we do.