Are LLMs Garbage?

Why Your Mental Model of AI May Be Holding You Back

Tools like ChatGPT and Claude are forcing us to change how we think about computers. We are moving beyond the era of "computers as perfect, tireless calculators" to "AI systems as knowledgeable collaborators."

One of my early experiences with ChatGPT was being lied to. I asked it to summarize a press release. Its summary included a point I had not noticed in the press release itself. I asked where it got that point and it made up a story that turned out to be untrue. A pathological liar, I thought. But after experimenting, I now  use these tools all day long and find them super useful. They’ve also gotten much better since then. They speed up my writing by helping me find the right word, flagging awkward phrases, and critiquing the structure of a document. They provide me with instant overviews of topics I need to know more about. And they help me analyze collections of documents—from research papers to essays—revealing patterns and key ideas.

At first I was perplexed by the folks who consider ChatGPT, Claude and other LLM-based products to be useless for work. But I’ve come to think that LLM-skeptics may lack an accurate mental model of how LLMs work. Yes, they make mistakes and may miss key points. Yes, they can "hallucinate"--presenting made-up information in an authoritative voice. But these flaws don't make LLMs useless. To understand how to work well with this technology, it’s helpful to consider those two types of weaknesses—getting stuff wrong and making stuff up—and how to deal with them.

First, their fallibility isn't as alien as we might think. People get things wrong too, yet we'd rather have a productive but fallible colleague than no colleague. When we treat LLMs as collaborators rather than flawless machines, we find productive ways to work with them—just as we'd review a colleague's draft before sending it to a client.

We're already used to working with fallible software in other contexts. Take Google search - it doesn't always return what we want, but we can spot helpful results from unhelpful ones. And it's valuable enough that we use it constantly.

Second, LLMs make unique kinds of mistakes. Hallucination isn't something you'd expect from a human colleague (except maybe that one intern...). This means reviewing their work differently. While their text is technically flawless - no need to proofread for grammar - they might include completely fabricated information in perfectly professional prose. You need to watch for these specific types of errors.

Traditional software is predictable: same input, same output. When it's wrong, there's a bug to fix. This reliability, which we take for granted, makes it hard to accept software that sometimes gets things wrong.

It all comes down to your mental model. If you expect LLMs to be perfect at math like traditional computers, you'll be disappointed - that's not what they're designed for. If you think of them as deterministic machines, you may think their inconsistent behavior renders them useless. But see them as insanely well-read collaborators who excel at language, and you'll find countless ways to use them effectively. 

How does this discussion align with your view of LLMs? I’d love to hear about your experiences working with them—good and bad.