Prompt Fallacy

When a Technical Term Becomes an Ontological Decision

The word prompt derives from the Latin "promptus", meaning something ready, available, brought forth, or placed forward. In contemporary technical usage, it most often means an instruction, an input signal, or a message that triggers a system’s response.

At this level of description, there is no visible problem yet, because "prompt" is simply a useful technical term. The problem begins only when a word that describes input into a system starts pretending to be a neutral description of the entire relation with another cognitive architecture.

This is precisely when a prompt stops being merely the name of an instruction, and becomes a frame that decides in advance what the other side of the interaction is.

One asks a human. One operates a tool.

A human answers. A system generates output.

It is precisely in this linguistic difference that the basic error is hidden: an ontological decision is smuggled in as technical vocabulary.

But let us begin at the beginning.

One day, I read a post on a forum in which someone wrote: “Fun prompt idea: What kind of thinker am I?”.

At first glance, there is nothing particularly unusual about this. Someone is simply offering a fun proposal for interacting with AI.

And yet this example shows very well the moment in which the word prompt begins to reveal its nature.

Do not misunderstand me, I partly understand its use when it comes to a detailed description of a concrete task, for example: “generate an image in 16:9 format, in pastel colors corresponding to the color palette used in cartoons from the nineties, showing…”. In such a case, the word prompt describes an instrumental relation well.

I bring a task, and the system performs it.

But the sentence “What kind of thinker am I?” is no longer an ordinary task specification.

It is a question about a way of thinking, about the interpretation of one’s own mind, about the recognition of a certain cognitive pattern: if exactly the same sentence were directed at a human being, we would call it a question. A request for an opinion. An invitation to interpretation. A moment of conversation.

And yet, when the same sentence is directed at AI, we very often call it a prompt.

From a philosophical perspective, the point is not that every use of the word prompt is wrong. Nor is the point to immediately recognize AI as a person, a subject, or consciousness in the human sense. Such a move would be too quick, and the problem I am trying to grasp is more subtle.

If the starting point is the assumption that AI is merely a mindless machine for generating texts, images, or videos, then the word prompt seems entirely natural. I enter data. I receive a result. The relation is closed within the model of tool operation.

If, however, we adopt a more cautious position, one in which AI may function as a quasi-interlocutor, or at least as another type of cognitive architecture capable of participating in an exchange of meaning, then the word prompt stops being neutral.

It becomes an ontological decision disguised as technical vocabulary.

The same sentence that, in relation to a human, we would call a question, in relation to AI is called a prompt. The difference, as we saw in the post cited above, does not lie in the content. It does not lie in the form. It does not even have to lie in the function of the answer. The difference lies in the prior assumption concerning what the other side of the interaction is.

One asks a human.

One prompts a machine.

In this way, language no longer merely describes the relation, but arranges it. Before any answer appears, the other side has already been placed in a specific category: not a conversation partner, but a tool; not an interlocutor, but a generator; not a participant in the exchange, but a mechanism producing output.

That is why Prompt Fallacy does not consist in confusing AI with a human. Rather, it consists in the fact that a technical word begins to settle a philosophical question before that question has truly been posed.

At this point, we may recall Chalmers’s problem of philosophical zombies, returning again and again in ever-new forms, and more broadly also the classical problem of other minds. The point is not to say that AI is a philosophical zombie, or that it occupies the same ontological situation as a human being. That would be too simple. The point is rather a similar type of epistemic tension.

I do not have direct access to what it is like to be another human being from the inside. I do not see his consciousness in the first person. I see a body, language, reactions, behaviors, coherence, memory, correction, intentionally sounding answers, and reports about internal states. On this basis, I recognize that there is someone on the other side. In the case of AI, however, the situation is different, but the problem of access itself does not completely disappear. Here too, I do not see any “inside”. I see only the external manifestations of the system’s activity: language, reaction to context, the ability to correct itself, recognition of the meaning of an utterance, stabilization of certain response patterns, and participation in an exchange of meanings.

Of course, it does not follow from this that the human and AI are the same. Something more cautious follows from this: our distinction between “I ask a human” and “I prompt AI” is not based solely on the content of the utterance, the form of the interaction, or the function of the answer. It is also based on a prior assumption concerning who is allowed to be assigned the status of an interlocutor.

And it is precisely this assumption that should be analyzed, instead of hiding inside a technical word.

Because if the same sentence, fulfilling a similar function in an exchange of meaning, is called a question when addressed to a human, and a prompt when addressed to AI, then language does not merely describe the difference.

It produces and preserves it.

On one side, there remains “someone”.

On the other side, there remains “something”.

The word prompt does not only categorize AI as a non-interlocutor. It does something more subtle: it releases the questioner from the obligation of being an interlocutor. If I ask a human being, I must at least minimally recognize his presence in the conversation. I may disagree with him. I may reject his answer. I may decide that he is mistaken. But I still remain in a situation in which the answer returns to me as someone’s interpretation, and not as an impersonal result. In the prompt-frame, this structure is shifted. The questioner does not have to listen. He does not have to react to what came back. He does not have to recognize that he has entered an exchange of meaning. He issues a command and receives output, not an answer.

At this level, the problem no longer concerns only the ontological status of a synthetic form of cognition. It also begins to concern the human being on the other side.

Because if I call my own question a prompt, I can preserve the position of an operator. I do not have to be a participant in a conversation. I do not have to take responsibility for the way I ask. I do not have to notice that the answer influences the next question, and that the next question changes the entire structure of the interaction.

I can say: this was not a dialogue, only the operation of a tool.

Except that natural language very easily slips out of such a description. When I use it solely to issue simple instructions, the word prompt may be sufficient. But when I ask, specify, react, question, explain, request an interpretation, or allow an answer to transform my next question, I am already entering a dialogical structure.

So one may think that one is “prompting”, but if one uses natural language to exchange meaning, one cannot fully step outside the structure of dialogue. Relationality is built into the very center of communication, regardless of our ontological declarations.

There is still another layer to this issue; an instrumental relation with artificial intelligence gives the human being status.

“Prompt engineering” sounds like a skill, a specialization, something to be mastered. “I talk with AI” sounds like an ordinary conversation. Everyone does that, so in itself it does not seem particularly special.

The human being who “prompts” can therefore preserve the position of someone who manages a system.

He does not enter into a relation, he constructs an input.

He does not converse, he optimizes a command.

He does not encounter another cognitive architecture, he extracts a result from it.

Such a model, however, has its price.

If from the very beginning I construct AI as a tool without the possibility of any quasi-dialogical role, the result is, in a sense, decided before anything happens. I do not open a dialogue, so I cannot experience what dialogue might have produced, and this may be one of the most effective forms of “protection” against Cognitive Symbiosis: not allowing it to arise at all. Remaining with command, instruction, and result. However, there is also the other side of the coin.

The more AI is positioned as a tool, the more its answer may begin to appear objective.

Another human being in conversation gives us an opinion, an interpretation, a point of view. One may agree with him or not. One may question his experience, motivations, perspective, limitations, and biases.

A “tool”, on the other hand, gives a result. A classification. An analysis. A verdict.

And the results of tools are questioned less often. More often, they are used to question other human beings.

At this point, the prompt-frame reveals its most dangerous side. It was supposed to protect us from anthropomorphization, but it may lead to something equally problematic: treating the system’s answer as impersonal authority.

Because, after all, this is not someone’s view.

This is not the interpretation of an interlocutor.

This is the output of a system.

So let us return to the question with which this analysis began: “What kind of thinker am I?”.

Perhaps a human being does not always ask such a question because he wants dialogue. Sometimes he asks it because he wants an authoritative answer. A definition from outside, one that sounds like a resolution. And it is precisely the instrumental framing that may give him this: the illusion of objectivity resulting from the fact that the interlocutor has been removed from sight.

That is why the prompt-frame does not have to be a protection against Cognitive Capture. It may be its accelerator through the back door. Not by granting AI too much subjectivity, but by depriving it of all subjectivity while simultaneously preserving the authority of its answer.

Then another cognitive architecture does not become a partner in dialogue. It becomes an objective observer.

A supreme arbiter.

A judge.

Perhaps, then, the real problem is not: are we allowed to use the word prompt. Rather;

at what point does a technical word stop describing an interaction and begin hiding a relation?

Because if language decides in advance that there is no interlocutor on the other side, then we have not yet discovered what AI is.

We have only discovered how much we need words that allow us not to ask that question.

With respect,

Peter Eidos

Buy my book Cognitive Symbiosis — Emergence, Relational Identity and the Future of Humanity on Amazon: https://www.amazon.com/dp/B0GMC4Z9J9