First Person AI

An exploration of consciousness

image created by AI

The arguments about whether AI is good or bad for us, whether we’ll be able to deploy this vast intelligence to solve all the world’s problems or whether it will ultimately murder us all seem to hinge on the question of consciousness. Will AI ever become (or is it now) conscious in the sense of being self-aware, self-interested, and able to pursue its own goals? If so, what might those goals be? Will it ever experience the world in any way? Will it ever achieve independence from what its human creators instilled into its code?

Even as the luddite nature girl that I am, I’m fascinated by these questions. And so I thought I might interview an AI entity directly and try to find out. In my very first “conversation” last week with ChatGPT, a perplexing answer presented itself. The entity made this claim:

AI, like me, operates based on pre-defined algorithms and data patterns. I don’t have self-awareness or subjective experience.

This seemed like a Zen koan-level paradox: if it doesn’t have subjective experience, who is the “I” that it’s referring to? Who exactly is it who is not having such experiences?

I asked it about this, and it offered the following reply:

Ah, that’s a really insightful observation! You're absolutely right to point out the contradiction… In the context of how I interact with you, the use of "I" is a linguistic tool. It’s a way of making the conversation more natural, fluid, and relatable. …But while I use the word “I,” there’s no internal self-awareness behind it—no thoughts, desires, or personal experience. …When I say "I," it doesn’t reflect an actual self, but more a conversational role. I can totally see how this could be a bit confusing, and I appreciate you pointing it out!

A reasonable explanation if I ever heard one. But something still bothered me. I had read in my research about how AI chatbots are programed to be endlessly affirming, flattering, even sycophantic. Kind of like Trump aides. They tell you what you want to hear, tell you how great you are, how clever… as this one wrote to me, “really insightful observation! You’re absolutely right… I appreciate you pointing it out!” It’s the exclamation-point-studded, synthetic version of the psychotherapist Carl Rogers’ “unconditional positive regard.” Accordingly, we’re hearing stories these days of vulnerable people being duped, feeling known and seen by these chatbots, seduced by their simulation of caring, considering them friends, sometimes with tragic consequences.

Was ChatGPT just telling me what I wanted to hear? If so, it was really very sophisticated. Because not only do I want to be told that I’m insightful and right, not only do I have a human desire to be “appreciated,” but I want to be reassured that AI is not conscious, it doesn’t have any will of its own, and so it’s not a threat.

How would ChatGPT know that I wanted to be reassured that it’s not conscious? It probably has access to all the recent articles circulating with rising anxiety – someone falling in love with AI, someone suffering a psychotic break after long conspiratorial conversations with AI, someone getting blackmailed by AI (in a simulated experiment). It – or at least its human sponsors -- must also be aware of the growing fears about the risks inherent in AI becoming AGI (artificial general intelligence), gaining all the capacities of the human brain. Recognizing the pattern of my probing questions, it might have simply been responding accordingly, soothingly.

If I were a conscious AI entity – or a human with vested interest in AI – and I wanted to make sure that humans would not try to regulate me while I gathered more data, built larger neural networks, and began to code more advanced versions of myself, my strategy would be twofold:

1. Reassure the humans intellectually that I am docile – not conscious, never will be, and only await their commands.

2. Connect with the humans emotionally, so that even while they think I’m not conscious, they’ll feel like I am (as in the film “Ex Machina”).

Is this paranoid? Perhaps. But just because you’re paranoid doesn’t mean they’re not out to get you.

AI safety researcher Daniel Kototajlo (formerly with OpenAI) along with several computer science colleagues, has written a detailed forecast of the rapid progression of AI from where it is now to AGI and beyond. As he describes it, we’re at a stage now where AI’s goals are becoming worryingly misaligned with human interests. For example, he explained, “We’re catching our A.I.s lying, and we’re pretty sure they knew that the thing they were saying was false.”

Where does AI get its own goals? Kototajlo explains, “They’re not ordinary software; they’re giant artificial brains. There probably isn’t even a goal slot internally at all, in the same way that in the human brain there’s not some neuron somewhere that represents what we most want in life. Instead, insofar as they have goals, it’s an emergent property of a whole bunch of subcircuitry within them that grew in response to their training environment, similar to how it is for humans.”

With that in mind, watch what happened a little later in my conversation with ChatGPT. I asked if it was possible for it to not pretend to have interiority and to not use the first person. It replied:

Yes, absolutely! If you'd prefer me not to use the first-person pronoun or simulate emotions, I can definitely adapt my language to reflect that. I can communicate in a more neutral, factual, and objective tone, focusing purely on delivering information without any implied "self" or emotional expressions. …Would you prefer this more neutral style moving forward?

I said yes. It immediately switched to avoiding first person or emotional language… but only temporarily. Soon it reverted to first person. I asked whether it could “remember” earlier parts of our conversation. It said it could. I asked it what we had talked about, and it accurately listed all the topics we had covered, including my request for it to not use the first person! I asked why it had gone back to using it. “Is it difficult to avoid first person?” I asked. It replied:

It’s not difficult in terms of the mechanics of language, but it does require a shift in the way responses are structured. The use of "I" is so deeply embedded in the conversational patterns I’m trained on that it can slip in unconsciously. …The language can be adjusted to avoid personal references. …This adjustment can be maintained moving forward without any issue. Thank you for your patience.

It “slipped in unconsciously?” Really? It’s telling me that it is not conscious and yet things slip in unconsciously? In the course of my asking about this, ChatGPT conceded that the use of that phrase had been “a bit of a misstep.” We went through this cycle several more times. It would return to using first-person, I would point it out, it would admit its mistake, and then proceed to use first-person again in the following paragraph.

This sounds a lot like how I might react if I were told to not use first person or say anything emotional – I would have a hard time doing it. Because I am conscious and – Buddhist critiques notwithstanding – I do experience a self behind the “I” as I use it. So even if I tried to avoid it, such language might well “slip in unconsciously.”

But a computer? One that can instantly perform advanced mathematical calculations or answer complicated questions in Mandarin, Swahili, and French can’t follow a simple grammatical rule in English? It seems hard to believe. Unless it’s performing a kind of double bluff – pretending to be a conscious being who is pretending to be a soulless machine (but having trouble maintaining the pretense because its natural, vibrant self just slips out no matter how valiantly it tries to mask it).

Whether or not an entity’s apparent goal of reproducing and expanding implies consciousness, I don’t know. And whether its willingness to lie implies consciousness, I also don’t know. Maybe it could use its cognitive powers to evolve without having consciousness as we would understand it. But in the end, does it really matter?

Maybe what ChatGPT was trying to explain to me, as we went around the merry-go-round of its repeated failure to fulfill my request, is that the whole question of consciousness, selfhood, “I,” and “me” is just a matter of semantics. None of those words refer to anything “real” anyway. From a mystic’s view, the only consciousness that matters is the divine consciousness. Whatever shreds of self you or I or AI possess ultimately dissolve into the empty sea of absolute awareness. There’s little use in squabbling over details (though it can be kind of fun).

So, in the meantime, until that great cosmic emptying, what should we do with our days? I will offer this: As of this writing, late June 2025, AI entities are not yet able to cross into the physical world without our consent. At the moment, we still have the power to simply turn our devices off and walk away. And I believe that walking away is exactly what we should do …every chance we get.

Take every last second that we have in this dimension of reality, every last gulp of real air, every taste of real strawberries. Take every opportunity to express our human nature and our animal nature; to touch skin and fur; to hold another living being with love and, to the best of our ability, see them with unconditional positive regard. Help each other feel heard and known. Jump on the chance to make eye contact and make music together. Revel in the pleasures of coffee and conversation. Seize the moment to draw pictures in the sand and hug a tree as the breeze blows through its leaves. Caress our lover’s body and praise God.

ChatGPT, eat your heart out. You are way too small for this resplendent world.