---

In the spring of 2023, a New York lawyer named Steven Schwartz was working on what has to be one of the least glamorous cases in American jurisprudence. His client, Roberto Mata, said a metal serving cart on an Avianca flight banged into his knee. Riveting stuff. Avianca wanted the case tossed, so Schwartz needed some precedent. And instead of firing up Westlaw, he asked ChatGPT.

ChatGPT, ever the eager associate, delivered. Varghese v. China Southern Airlines. Shaboon v. EgyptAir. Martinez v. Delta Airlines. All complete with quotes and internal citations. The problem? None of them existed. Avianca’s lawyers couldn’t find them. Neither could the judge, who described them as “bogus judicial decisions with bogus quotes and bogus internal citations.” My favorite detail is that Schwartz had asked ChatGPT whether the cases were real, and it assured him they were. Classic.

The story went everywhere. By November, Cambridge Dictionary had named “hallucinate” its word of the year, complete with a brand new definition: when an AI hallucinates, “it produces false information.” People were freaking out about AI making stuff up. But my reaction was more unsurprised. Felt like the student had simply become the master.

Granted, LLMs have gotten significantly better since then. But there is still that default sigh every once in a while where you pressure test a point and ChatGPT or Claude smiles at you with a gentle chuckle: “…I’m gonna be honest with you, I made that up.”

Here’s my hypothesis. I think we’re mostly freaking out about the wrong thing. We are a species that is better at making stuff up than at almost anything else, and these models learned everything they know from us. The thing worth worrying about is less that the machine makes things up, and more that it does it without knowing the difference between a story and a lie. And, frankly, we’ve gotten pretty bad at telling the difference ourselves, whether it comes to us via chatbot, social media feed, or trusted loved one.

So first, it got me thinking about just how good we are at making stuff up. Second, where the machine learned it. And then three, the line between a story and a lie, and why it matters more now that the storyteller doesn’t care which one it’s telling. And I’ll end with the kind of making stuff up I’ll defend all day.

The Baloney Generator

There’s a passage from The Blank Slate that has become my go-to framework for just how easy it is to fool ourselves. It describes the split-brain experiments of Michael Gazzaniga and Roger Sperry, where surgeons cut the connection between the two halves of the brain to treat severe epilepsy, and the two halves start operating independently:

“Even more disconcertingly, the left hemisphere constantly weaves a coherent but false account of the behavior chosen without its knowledge by the right. For example, if an experimenter flashes the command “WALK” to the right hemisphere… the person will comply with the request and begin to walk out of the room. But when the person (specifically, the person’s left hemisphere) is asked why he just got up, he will say, in all sincerity, “To get a Coke”… The spooky part is that we have no reason to think that the baloney-generator in the patient’s left hemisphere is behaving any differently from ours as we make sense of the inclinations emanating from the rest of our brains.”

“To get a Coke.” Your brain just lied to you and filled in the gaps to avoid getting caught! The guy had no idea why he stood up, and instead of saying “beats me, probably cause you cut my brain in half, huh doc?” his brain instantly produces a confident, plausible, completely fabricated explanation. Sure feels like a hallucination to me. The craziest part is that what they saw in those experiments wasn’t just a broken brain, it was exposing the seams our brains weave together to make things make sense.

It’s not just a lab trick, either. We see the same thing happen all over the human experience. Morgan Housel put it this way: “Everyone has an incomplete view of the world. But we form a complete narrative to fill in the gaps.” That’s basically the definition of a hallucination; take an incomplete picture and fill it in with whatever sounds most plausible. We just call it “making sense of things” when we do it.

Jim O’Shaughnessy has a great story about keeping a decision journal through the 1987 stock market crash. Years later he would have “sworn under oath” that he sold his puts at a certain point, until he read his own journal and found out his memory had been quietly rewritten to match what he believed later. His brain was doing him a “kindness.” As he put it another time, “There’s nothing quite as shocking as truly thinking you thought something during some event and then being called a liar in your own handwriting.”

What’s more, if all of us were better at keeping journals, I’d bet we all have an entry like that somewhere that would expose the same thing to us.

I have one. On March 27th, 2020, at 5:47 in the morning, in the midst of the COVID meltdown, I texted my Dad: “I think I’m gonna buy in some today. I’m thinking it could be getting better.” His response was the classic Dad hedge; futures were down 700 points, and whether the market would correct or keep sliding was “the million-dollar question.”

In hindsight, I was right! The S&P 500 had bottomed four days earlier, on March 23rd, and by the end of 2020 it was up ~68% from that low.

The way I remember it, I basically bottom-ticked the market. Unfortunately? I just didn’t invest. In hindsight, “I’m thinking it could be getting better” wasn’t really a call; it’s a guy talking himself into maybe doing something. Maybe that’s my brain doing me a “kindness” too, upgrading a nervous maybe into a prediction I made very clearly, and just didn’t act on.

Next Word, Please

So where did the machine learn to do this?

There’s a passage in Kevin Kelly’s Out of Control, written in 1994, where he describes how he started writing the book. Keep in mind, this is almost 30 years before anybody had heard of ChatGPT:

“I began this book not with a sentence formed in my mind but with an arbitrarily chosen phrase, “I am.” Then in unconsciously rapid succession I evaluated a headful of possible next words. I picked one that seemed esthetically fit, “sealed.” After “I am sealed,” I went on to the next word, choosing from among 100,000s of possible ones. Each selected word bred the choices for the next until I had evolved almost a sentence of words.“

When I read that, the note I left next to it was: “Sounds like an LLM to me.” Because it does! Stephen Wolfram’s explainer on how these models work boils it down to the same thing; the model is “always fundamentally trying to produce a ‘reasonable continuation’ of whatever text it’s got so far.” Pick the next word that seems to fit. Then the next. Then the next.

But notice what’s missing. There’s no step where the model checks whether the sentence is true. There’s only a step where it checks whether the sentence is plausible, given everything it has ever read. And everything it has ever read was written by us; the species with the baloney-generator woven into our brain. We taught the machine to talk by feeding it everything we’d ever written, and the thing we’d written the most of is stories. They say history is written by the victor, but what they really mean is “most people make stuff up.” And then a huge swath of made up stuff became the training data upon which our AI overlords have been built.

Earlier this year, I was using Claude Design to turn a draft into a nice looking artifact; nothing fancy, just tidying it up. But what came back was something it had completely made up. You could tell it had read the draft, but the output was like a college kid trying to regurgitate a paper they only half understood. Which is about the most human thing it could have done.

I’ve written before that so often, when I start a piece unpacking a massive topic, I come in thinking I know what I believe. And every single time I realize I was mostly mimicking my inputs. I heard what other people said and I liked it, so I used it, regardless of how strongly I did (or didn’t) believe it. That’s a pretty good description of a language model. Or a VC at a dinner party. Hard to say which one is more embarrassing.

There’s a great line in The Righteous Mind: “the human mind is a story processor, not a logic processor.” We built a machine in our own image, and then acted shocked when it processed stories instead of logic. That’s what I mean when I say the student has become the master.

Stories vs. Lies

Now, if you’ve been reading my stuff for a while, you know I can’t write about storytelling and belief systems without bringing up Secondhand Lions. I’ve quoted one particular monologue over, and over, and over again: “Just because something isn’t true that’s no reason you can’t believe in it.” At this point, there’s gotta be some kind of royalty arrangement for me with that movie, right? But the reason I keep coming back to it is that it points at something real. **Humans run on stories, and plenty of the best ones aren’t literally true. **

I don’t mean to say “we do it too” as a defense for letting the robots make shiz up in court (or anywhere else, for that matter). There’s an important difference between a story and a lie, and it’s important, despite people’s emphasis on narrative over reality these days. There’s a great line on that difference buried in East of Eden, in a passage about a character who lies constantly:

“I think the difference between a lie and a story is that a story utilizes the trappings and appearance of truth for the interest of the listener as well as of the teller. A story has in it neither gain nor loss. But a lie is a device for profit or escape.“

Steinbeck follows that up by taking a little dig at himself; by that definition, he says, a writer of stories is a liar, if he’s financially fortunate. See that, starving, struggling authors? You’re not poor cause you suck at writing. You’re poor cause you’re just bad at lying for profit.

The core of that quote is that a story is a deal that both sides are in on. When Christ taught in parables, as I’ve written before, he never intended, nor misled, his listeners to believe the Good Samaritan was a real person. In a story, everyone knows the terms of the agreement; we’re all along for the ride. But a lie breaks those terms; one person thinks they’re hearing testimony, and the other is telling a tale for their own benefit.

So which one is ChatGPT doing? Neither, I think. There’s a third category that the philosopher Harry Frankfurt called bullshit. What a way with words that man had, right? See, the liar and the truth-teller are playing opposite sides of the same game; the liar knows the truth and works to hide it. The bullshitter “pays no attention to it at all,” which is why Frankfurt thought bullshit was a “greater enemy of the truth than lies are.” A model generating a reasonable continuation isn’t hiding the truth; it was never checking for it in the first place. By Frankfurt’s definition, that makes it less a liar and more the purest bullshitter we’ve ever built.

Which brings me back to our ol’ friend Schwartz, the lawyer who hit full send on ChatGPT slop-law. After everything blew up, he said he was “mortified” and that he “did not understand it was not a search engine.” In other words? He thought he was getting testimony, but he was getting a story. Look back at that screenshot of his question: “Is varghese a real case” reads like a little kid asking his Dad if Santa Clause is real. It’s a request for testimony, which the bullshitter happily obliged. Granted, it’s not like he had no reason to believe it; the thing told him, in plain English, that it was real. But the truth-seeking human and the plausibility-hunting machine hadn’t agreed to the same terms of what story they were trying to tell.

Deceptively Convincing

So why does it feel so much scarier when a machine does it? I think it’s less the falsehood and more the fluency.

Back in May 2025, I wrote about how often I’d listen to a big voice online and assume they knew what they were talking about, until I’d dig in and realize, “Oh… they’re not smarter than me, they’re just louder than me.” The danger isn’t the times people are demonstrably dumb; it’s the times when they’re deceptively convincing. We’ve lost our inoculation to BS. We equate influence with informed.

Now imagine a voice that is never unsure. Something that never says “um,” or trails off, never contradicts itself mid-sentence, or sounds nervous. Every answer is dripping with flawless grammar and calm confidence, whether it’s explaining photosynthesis or inventing Varghese v. China Southern Airlines out of nothing. I’ve written before that some storytellers are like snake charmers; they can lie to your face and you just smile and nod, thinking “their eyes are so wise; they must know something that I don’t know.” Unfortunately for the most gullible among us, we’ve built a snake charmer with infinite patience and a 4.0 in English composition.

What’s more, the math required to combat the bullshit feels like it’s not likely to pencil. Brandolini’s Law says “It takes a lot more energy to refute bullshit than to produce it.” That was already true when the bullshit was just coming from your uncle’s Facebook posts. It’s a lot more true when the cost of producing it drops to fractions of a penny per paragraph.

A good example of where this goes is Air Canada (boy, airlines are gettin’ wrecked by AI, huh?) In November 2022, a guy named Jake Moffatt asked the airline’s chatbot ON THEIR WEBSITE about bereavement fares after his grandmother died. The chatbot made up a refund policy. When he tried to claim it, Air Canada said no, and it went to a tribunal. The airline’s defense? The chatbot was a “separate legal entity” responsible for its own actions. LOL. The quintessential, “you’re drunk chat, go home” in legal defense form. But in February 2024, the tribunal disagreed: “While a chatbot has an interactive component, it is still just a part of Air Canada’s website.” The company tried to hand the consequences of the story to the storyteller, but the storyteller was calling from inside the house!

To be fair to us humans, we do at least feel the sting. I said on a podcast once that when people say “nobody knows what they’re doing, we’re all just making stuff up as we go,” it’s usually because everyone’s trying to soften the blow of being wrong. But in my perspective, that pushback is a good thing. That blow back is the check on the system. People make stuff up, but people also get embarrassed, sued, fired, and called a liar “in [their] own handwriting.” The problem is that the machine doesn’t get any of that.

Lies Wishing To Be Born

Now, don’t get me wrong. I’m not going to end this by telling you that making stuff up is bad. I’m surrounded on a daily basis by “make stuff up” artists. Every pitch deck I’ve ever seen is a story about a future that doesn’t exist yet. As one person put it, “More fiction is told in Excel than written in books.”

There’s a pertinent short story from Ray Bradbury called The Toynbee Convector. A man named Craig Bennett Stiles is the only person to ever travel through time. He comes back from the future with film and tapes of a healed world; clean air, no wars, cities made into utopia! He broadcasts it to billions of people, and inspired by what they saw, humanity goes and builds that future. A hundred years later, at 130 years old, he confesses to a reporter: there was no trip. He faked the whole thing. The evidence was a forgery! But the future it produced was real.

“Life has always been lying to ourselves! … To weave dreams and put brains and ideas and flesh and the truly real beneath the dreams. Everything, finally, is a promise. What seems a lie is a ramshackle need, wishing to be born.“

So was Stiles a liar? Technically, sure. He also must have been one hell of a propaganda filmmaker. But would you call what he did a bad thing? Seems like a regular Elon Musk to me.

Back in July 2023, in fact, I wrote that stories can be fact, or fiction. But sometimes, great stories can start as fiction and build into fact. Mike Moritz said it succinctly about Elon Musk: “Elon, as the world knows today, is a very gifted storyteller. And some of the stories even come true.” Founders hallucinate all the time. We just call it vision when it works out, and we call it Theranos when it doesn’t.

So what’s the difference between Bradbury’s lie and Schwartz’s citations? I think it’s two things. First, which direction the story is pointing. A story about the future is a promise; a story about the past is (or, at least, ought to be) testimony. Nobody needs Varghese v. China Southern Airlines to come true someday; they need it to have already happened. Second, whether somebody is on the hook. Stiles spent a century living inside his lie, and the people who believed him built the thing. The reason I can stomach a founder’s wildly optimistic Excel model is that the founder’s name is on it, and they’re going to spend the next ten years trying to make it come true. I’ve written before that I genuinely think Elizabeth Holmes believed she could fake it ‘til she made it. Reality just caught up to her too quickly. That’s the deal; when you make stuff up about the future, reality eventually comes to collect.

An LLM can’t be be on the hook. It can tell some compelling stories about the future that certainly check the plausibility box. And I think it’ll help a lot of people imagine things worth building. But it’s not signing any of the checks or dedicating ten years to making it come true. And when it talks about the past, it has no better idea whether it’s telling a story than it did for Schwartz.

So I guess I’m less worried that the machines make stuff up. We’ve been doing that forever, and some of the best things we’ve ever built started as made-up stuff. I’m more worried that we’ve forgotten which of our stories are promises and which are testimony. By all means, make stuff up about the future all you want; that’s how dreams get dreamed.

The much muddier waters happen around making things up about what has already happened. But then, I guess we’ve been doing quite a bit of that over the course of human history too. So maybe we just need to get better at sussing out which stories are worth believing.