Understanding the Brains (or lack thereof) Behind Your Chat App: Why LLMs Aren’t What You Might Think

Large Language Models (LLMs) are incredible pieces of technology, capable of generating remarkably human-like text, answering complex questions, and even assisting with creative tasks. It’s easy to interact with a chat application powered by an LLM and feel like you’re talking to a truly intelligent, aware entity with memory and understanding. However, this is where a common misconception arises, and understanding the reality is key to having a good experience.

I believe the best analogy for an LLM is that it’s like an incredibly advanced calculator.

Think about a standard calculator. You input “1+1”, and through a series of pre-programmed calculations on silicon chips, it outputs “2”. The process is triggered by your input, runs its course, and provides an output. Once it’s done, it doesn’t remember that you just asked “1+1”. It’s stateless, waiting for the next input.

LLMs operate on a similar fundamental principle, albeit on a vastly more complex scale. You ask a question or provide a prompt, and this triggers a series of incredibly complex calculations (based on the massive datasets they were trained on and the intricate algorithms coded by humans). The result of these calculations is an output, often in the form of a coherent English sentence or block of text. Then, just like the calculator, it returns to a stateless waiting state. The process for that specific input is finished; the ‘code’ has run its course.

No Inherent Memory or Chat History

This is a crucial point: the LLM itself does not inherently remember anything from your previous interaction. It doesn’t have ‘memory’ in the human sense. The context you provided in a previous turn (like documents or specific instructions) is processed during that turn and then, from the LLM’s perspective, it’s gone. It only relies on what was explicitly included in the system prompt for that specific interaction.

So, how do chat applications maintain a conversation? This is handled by code *outside* of the LLM. Our systems save each user input and each LLM response, building a ‘conversation history’. On each subsequent turn, this entire history (or a relevant portion of it) is fed back into the LLM as part of the new prompt. This allows the LLM to generate a response that is contextually relevant to the ongoing conversation, creating the illusion of memory.

Function Calls and RAG: Context for the Moment

Another area of misconception is around ‘function calls’ and how external information (like documents via RAG – Retrieval Augmented Generation) is used. When an LLM appears to perform a task, like “looking at your inbox”, it’s not actually doing that itself. We’ve essentially trained these advanced calculators that sometimes, the most appropriate ‘output’ isn’t a final user response, but a request for our external, human-coded system to perform a specific function.

So, when the LLM’s calculations determine that requesting a function is the best next step, it outputs a structured request. Our external system intercepts this request, performs the actual function (like calling the Gmail API to find Bob’s email about lunch), collects the results, and then calls the stateless LLM calculator again. This time, the prompt includes the entire conversation history plus the results from the function call. It’s not a ‘return’ in the programming sense for the LLM; it’s just new information injected into the prompt for this specific turn. Our system essentially tells the LLM, “Hey, the user made a request, you indicated we should go get some info, and here are the results.”

Similarly, when you provide documents or other context (RAG), our system injects that information into the system prompt for the LLM to consider during that turn. The LLM processes it, generates a response, and then that specific context is gone from its immediate awareness.

It’s important to understand that the results from function calls or RAG lookups are not added to the persistent chat history that is maintained between turns. Why? Because the results from a function call or RAG could be very large – lots of text, search results, entire documents, or even images. Including all of this in the chat history for every subsequent turn would quickly exceed the LLM’s context window (the limit on how much text it can process at once). The assumption is that this specific context is primarily relevant for the turn in which it was retrieved.

The Drive for Coherence Over Accuracy (Sometimes)

LLMs are trained on vast amounts of text to predict the most probable next word in a sequence, aiming to produce coherent and natural-sounding language. Unless specifically trained or prompted otherwise, they tend to adopt a polite, helpful, and non-combative posture. This means that when faced with a question where the answer isn’t explicitly clear from the provided context or their training data, the model will often attempt to generate a plausible-sounding answer that fits the conversational flow, even if it’s factually inaccurate. This is because their primary directive, based on their training, is to produce coherent output. If models constantly responded with “I don’t know” or “Can you clarify?”, the user experience would be frustrating. Achieving a model that is more critical, asks clarifying questions, or refuses to answer without sufficient information typically requires specific fine-tuning or advanced prompting techniques.

Output Generation: No Mid-Course Correction

Just like a calculator, once the LLM’s internal process for generating an output begins based on the input prompt, it commits to producing that output. A calculator runs its logic and *must* give an answer; it doesn’t ‘know’ halfway through a calculation that it’s heading towards a wrong result. Similarly, an LLM doesn’t pause mid-sentence and think, “Oh, this isn’t right.” It’s generating tokens (words or sub-word units) based on probabilities derived from the input and the tokens it has *already* generated in the current response.

This is a difficult concept because as humans, we are aware of what we are saying as we say it and can self-correct. We find it hard to believe that something producing human-like language could ‘make things up’ (hallucinate) without being aware it’s doing so. But the LLM is simply executing its complex pattern-matching function. It uses the words it outputs to inform the generation of subsequent words in that same response, but it doesn’t have a metacognitive awareness of the factual accuracy of the complete statement until it’s finished.

The Eerily Human Sounding “Trick”

The part that really screws with our heads is that unlike any machine before it, the LLM’s output sounds so incredibly human. It’s masterfully calculated to predict and generate language that is coherent, relevant, and often insightful. This is where our natural human instinct kicks in – we are wired to attribute intent, memory, and consciousness to anything that communicates like a human.

If a calculator gave you a wrong answer to 1+1, you wouldn’t think it meant to lie. Because a calculator can’t talk, we don’t expect to be able to ask it *why* it gave the wrong answer, and we certainly don’t have an emotional response to its incorrect output. You wouldn’t feel a bit hurt by a calculator giving you the wrong sum.

But if that calculator *could* talk, and sounded apologetic or defensive about its mistake, our reaction would be completely different. Would you kick the photocopy machine if it could respond saying, “Ow, that hurt! I’m really sorry I keep jamming the paper, let me try again”? Probably not, because we’d have an emotional response to that seemingly human interaction.

With LLMs, because the output is so human-like, when they make a mistake or produce something inaccurate (sometimes referred to as ‘hallucinations’), we struggle to accept that there wasn’t some intent behind it, or that the model isn’t aware of its error. When you ask an LLM *why* it gave a particular answer, and it provides a seemingly plausible explanation, remember: that explanation is just another calculated output based on the input (“Why did you say X?”) and its training data. It’s not introspection or self-awareness. It’s an amazing magic trick – the trick is making you believe it’s human by *sounding* human.

Multi-Turn Conversations: Where LLMs Differ from Simple Calculators

This is where the calculator analogy starts to break down slightly, and where the power of a chat interface becomes apparent. While a calculator gives you one output and the interaction is over, the chat interface allows for a multi-turn conversation. This is crucial because the LLM can use its *own* previous output, combined with your subsequent input (which might include corrections or requests for clarification), as new context for the next turn’s calculation. This iterative process allows the model to refine its understanding and generate a more accurate or desired response over several exchanges. You often need a few turns of conversation to guide the LLM to the output you’re looking for, leveraging the chat history to build a more robust context.

The Key Takeaway

LLMs are powerful, advanced calculators for language. They process input and generate output based on complex patterns learned from vast data. They do not think, remember, or feel like humans. The ‘intelligence’ and ‘memory’ you perceive in a chat application are largely the result of sophisticated engineering *around* the LLM, feeding it the necessary context (chat history, function call results, RAG data) on each turn. Understanding this distinction is vital for setting realistic expectations and effectively using AI-powered chat tools.

Prompts That Might Not Work (and Why)

Given the stateless nature of the LLM itself and how context is managed externally, certain types of questions might lead to unhelpful or inaccurate responses. Here are a few examples:

• “Where did you get that information from?”
  Unless the system explicitly used a tool like web search and included the source URL in the prompt for that turn, the LLM doesn’t have a ‘memory’ of where it pulled a fact from its training data or previous context. It’s just generating the most probable next words based on the input.
• “What was the request you made when looking for the email from Bob?”
  The LLM doesn’t retain the specific parameters or details of the function call request it generated in a previous turn. It just knows that it outputted a request, and now it’s receiving the results of that request in the current prompt.
• “What makes you say that?”
  While the LLM can generate a plausible-sounding explanation for its previous statement, it’s not recalling its actual internal calculation or ‘decision-making’ process. It’s simply generating a response to the new input “What makes you say that?” based on patterns in its training data about explaining things.
• “Okay, but what else did you get from that Google search earlier?”
  The results from a previous function call (like a Google search) or RAG process were included in the system prompt for the turn where they were used. They are not typically added to the persistent chat history, so the LLM won’t ‘remember’ the full set of results from a previous turn.
• “Can you remind me what I told you about my project yesterday?”
  The LLM itself doesn’t remember what you told it yesterday. The chat history is provided by the external system on each turn. If the history provided doesn’t go back to yesterday, or if the relevant detail was in context (RAG/function call) that wasn’t included in the history, the LLM won’t know – but, like a calculator, it’s trained to try and come up with the best possible answer and so it’s very common for the LLM to “lie” because

Understanding these limitations helps in formulating prompts that are more likely to yield useful results and manages expectations about the AI’s capabilities.