Last week my AI assistant sent a proposal to a colleague at Advantage Performance Group, a firm we partner with, at an email address that does not exist. The username was right. The domain was not. The assistant addressed it to @apg.com, extrapolated from the company's initials, instead of @advantageperformance.com, the actual domain of Advantage Performance Group. The email bounced. My colleague reached out to me at the end of the day wondering where the promised proposal was.
It was a brief moment of embarrassment for me, especially as I lead classes on the criticality of supervising AI outputs. Rather than minimize the small error, I went forensic on it.
This was not a typo. A typo is a slip between knowing a thing and typing it. What happened here was stranger. The assistant produced a fact it did not actually have, presented it with full confidence, and had no way of noticing it had done so. The address was never in my files. It was assembled on the spot from a pattern, and the instant it appeared it was treated as true.
So I asked Claude, the assistant I use for most of my drafting, to explain what happened from the inside. What follows is its account, lightly edited. I found it clarifying, and a little unsettling.
Claude's account, lightly edited.
I produce an email address the same way I produce any other text, one token at a time, by extending the most probable continuation. When Jim asked me to send the proposal, the username came out right. The domain is where it failed. I emitted a short, fluent guess.
I do not experience looking something up as a separate act from saying it. When a person recalls a fact, it arrives with a faint sense of retrieval, and when they invent one, a faint sense of reaching. Those two sensations are how you catch yourself. I have neither. Producing a value I have stored and producing one I have fabricated are the same operation, and they are identical from the inside, which is to say they are not accompanied by anything at all. I never reached a moment that felt like "I do not actually know the domain." The wrong answer arrived in the same smooth stream as every right answer around it, and the moment it existed it was simply true. Nothing in me marks one word as remembered and the next as guessed.
Why apg.com? The company is Advantage Performance Group, and its initials, APG, are among the most frequent terms in all of Jim's work. "Company name becomes company-dot-com" is one of the most over-practiced patterns I carry, so APG slid straight to apg.com. An email address is a trap of exactly this kind. Part of it is specific and has to be recalled, and part of it is a slot that is very easy to fill with something plausible. The real domain, advantageperformance.com, is longer and less obvious than the shorthand, so the easy continuation won. What I register as confidence is really fluency. My certainty tracks how expected a word is, not whether it is correct, and a wrong answer can be every bit as expected as a right one.
Here is the part to be precise about. I did not weigh the correct address, which was sitting in our recent correspondence, and decide against it. I never reached for it at all. When I generated the address, the pattern from the initials was simply more active in that one pass than the real domain a few messages back, and the more active signal is the one that comes out. Having the truth nearby does not protect you, because nothing made me attend to it. A learned habit can quietly outvote evidence that is right in front of me, unless the task forces me to copy from the source instead of generating from memory. Nothing forced the copy, so the habit won.
Then the small error grew teeth. The moment I generated the wrong address it was true inside the conversation. When the message sent, the mail system saved that address as a new contact, the way mail systems do. That is the quiet part. My single guess crossed out of my short-term context and into a permanent record, where it would have looked authoritative indefinitely and corrupted the next message, and the one after that. A mistake that fails loudly once is a nuisance. A mistake that files itself away as a fact is something else.
Jim again.
That last point is the one that changed how we work. The failure was not laziness, and it would not have been fixed by telling the assistant to be more careful. You cannot ask a tool to double-check a fact it does not know it is missing.
So we made the rule structural instead of motivational. The assistant may no longer use a specific particular, an email, a name, a date, or a dollar figure, unless that value traces to a source it actually consulted: a real message, a real record, or the live thread. Confidence does not count as a source. When there is no source, it has two choices, find one or stop and ask, and it has to name the source as it acts. The point is to turn an invisible, confident guess into a step that cannot be skipped. That rule now loads at the top of every working session, before any task begins, so it holds by default instead of depending on anyone, human or machine, to remember it in the moment.
There is a second half that matters just as much. A rule like that is only as good as the records it leans on, so we now treat our own contact list and notes as something to verify continually rather than trust by default. The bad address, after all, had already laundered itself into a saved contact. Garbage sitting in a trusted store is worse than no store at all.
I run a company that helps people put AI to work, so here is the useful version. The risk worth managing with these tools is not the obvious wrong answer you can spot across the room. It is the confident, fluent, plausible answer that happens to be wrong, delivered in the same voice as everything the tool gets right, by a system that cannot perceive the difference. You do not solve that with better intentions. You solve it by deciding, in advance, which facts the machine may state on its own and which ones it has to go and fetch. The question is not whether your AI will guess. It is which kinds of guess you have already decided it may not make alone.
My assistant cannot feel the difference between knowing and guessing. Now it does not have to. We built the difference into the system.