AI Unplugged: Smart Enough

Generative AI is everywhere, and a lot of it isn't great. But what happens when it's so good we can't tell the difference?

I've experimented with 10 different AI platforms, each more human-like than the last. The frontrunner is now Anthropic's Claude 3, followed by Inflection's Pi. And the truth is, they're fast becoming indistinguishable from text-based conversations with real humans on the Internet. So perhaps it's not a surprise that if they can chat like people, they certainly can write like people. And if they can write like people, they will begin intruding in the space where humans traditionally work, like school, branding, marketing, and creative writing (my day job!).

How Smart is Smart?

It's an interesting question. What makes a generative AI "smart?" Perhaps the better question is: is the AI smarter than the bulk of humanity? The average IQ is 100, with a lower range of 85 and an upper range of 115. So for AI to be comparative to humans, it doesn't need to be brilliant -- it just needs to be smarter than us.

You can see how smart AI is, on average, ranked by IQ by Maxim Lott. Using average humanity as a base, there are already two AI in human range: ChatGPT-4 (85) and Claude-3 (101). So if there were concerns about generative AI being indistinguishable from humans on some tests, we're already there.

We know that some AIs are "smart" -- intelligence can be measured many different ways -- and emotional intelligence (EQ) is often how we can detect where AI fails. But what happens when AI advances to the point where we can no longer tell the difference?

The battle of AI detection has reached a fever pitch in academic institutions, where writing quality is a measure of performance. And thus we come to the Poetry Test.

"I Think That I Shall Never See, an AI As Lovely As a Tree..."

Sierra Elman asked three generative AI (ChatGPT-4, Google's Bard (pre-Gemini), and Anthropic's Claude-2) to write a poem. They were judged by 38 AI experts and 39 English experts to answer the question: is an AI smarter than an 8th grader?

The answer? Not yet:

Most strikingly, English experts were far better at discerning which poems were written by AI, with 11 English experts vs. only 3 AI experts guessing the author (human vs. AI) of all four poems correctly. This points to a need for English experts to play a greater role in helping shape future versions of AI technology.

Of the three, humans performed at 69%, followed by Bard (62%), Chat-GPT (58%) and Claude (54%). Mind, this was Claude-2; I suspect Claude-3 would do much better in this test.

The AI Student

The results of that test matters a lot. Teachers, who previously waged a battle to detect Wikipedia-copied content now have to contend with an amorphous, real-time, human-sounding artificial intelligence.

Poetry in particular is an interesting test because it requires more creativity; essays that are about history, articles about facts that don't require much effort on the student, can easily be created convincingly with AI, assuming it doesn't hallucinate. More to the point "bad writing" -- the kind that as my one English teacher termed "scarf and barf" -- is something AI can already replicate relatively easily. But then, that was never great writing in the first place, and using that method as a means of grading students is something we should probably strive to move away from as a learning model.

But until we do, AI is going to be a real problem. Humans are notoriously bad at detecting falsehoods, and AI speaks with great authority.

Part of the issue with AI is that the Internet does not allow for our usual tools: in-person physical cues, voices, facial expressions (my Master's thesis was based on this very topic). To compensate, our own biases have a stronger influence in how we receive information from others. AI capitalizes on this weakness, providing not multiple answers but just one. The capacity for bias -- not just on the human side, but on the AI's side originating with its creators -- is enormous.

Add all this up and AI doesn't have to be a genius to be an effective writing partner, spread disinformation, or upend academic institutions everywhere. It just has to be better than you and me. And for a large bulk of humanity, it already is.

Please Note: The views and opinions expressed here are solely my own and do not necessarily represent those of my employer or any other organization.