Looks like a lawyer wrote it
People are taking legal advice from a chatbot. Signing contracts it wrote. It looks right, nobody checked it, and that’s the whole problem.
I keep meeting people who take legal advice from a chatbot. Not a lawyer. GPT. Claude. A few of them are working off contracts the thing drafted for them.
And it looks right. The legalese is clean. The structure is convincing. It reads like something a professional would hand you across a desk.
Nobody checked a word of it.
That’s the trap. The danger was never that AI gets things wrong. Everybody knows it gets things wrong. The danger is that it’s convincing when it’s wrong. It hands you a clean, confident, finished-looking answer, and finished-looking is what buys your trust.
Compelling and correct are not the same thing.
So people give it trust it never earned. They read the fluent paragraph, it sounds like authority, they stop asking whether it’s true. Then they sign it.
And a wrong answer that looks right is the worst kind. The clean one gets believed, written down, and from there it compounds.
The more time you spend in these tools, the less you trust the answer that looks finished.
That’s me, from living in them every day. Not because the tools are bad. Because I’ve watched too many clean, confident answers fall apart the second I checked them.
So the verifying is the job now. Knowing what to distrust, where it drifts, what to check before you sign it or send it or act on it. Everyone’s skipping that step.
And that job doesn’t shrink as the models get better. It grows. The more sophisticated the tool, the more finished the wrong answer looks, and the more it takes a human to catch it. You’re the orchestrator. Especially now. Especially as this stuff gets good.