Skip to content

Can you catch the AI making things up?

AI sounds just as sure when it is wrong as when it is right. AI literacy means knowing how it works well enough to spot the difference, and being willing to say “that is wrong”.

Spot the made-up answer

Round 1 of 4. One of these is made up. Tap the one you think is fake.

Both answers always sound the same. That is exactly what makes AI mistakes hard to spot.

What AI is really doing

You do not need to build AI to understand it. Four ideas are enough to know when to trust it and when to check.

It makes guesses

Not magic, and not a mind. It predicts what comes next, using what people gave it to learn from.

It can make things up

When it does not know, it can still give an answer that sounds perfect. Nothing in its voice changes.

It slips in the same places

Exact facts, numbers, sources and recent news are where to look twice.

You can say it is wrong

Disagreeing with something that sounds so sure takes a little courage. That courage is part of the skill.

1997

the year it got a name

Trusting machines too much is not a new problem.

“Misuse refers to over-reliance on automation, which can result in failures of monitoring or decision biases.”

Decades before today’s chatbots, researchers studying pilots and others working with automatic machines noticed this: when a machine is usually right, people stop checking it, and its mistakes become theirs. The fix is a habit anyone can learn.

Parasuraman & Riley, Human Factors, 1997

Real case files

Scientists put AI to the test. Here is what they caught.

Case file 1Caught

The books that do not exist

55%

ChatGPT-3.5

18%

ChatGPT-4

Researchers asked ChatGPT for short reports with references, then checked all 636 of them. That is how many were invented: titles that looked real but were never written. Newer versions made up far fewer, but still some.

Clue: Check that a source actually exists.

Walters & Wilder, Scientific Reports, 2023

Case file 2Caught

The answer that felt right

Longer answer, confidenceUp
Longer answer, accuracyNo better

People read AI answers to quiz questions and guessed how likely each was to be right. They thought the AI was right more often than it was, and longer explanations made them more sure, even when the answer was no better.

Clue: Long and confident is not the same as correct.

Steyvers et al., Nature Machine Intelligence, 2025

Case file 3Caught

Even experts get fooled

Newest doctors

AI right79.7%
AI wrong19.8%

Most experienced doctors

AI right82.3%
AI wrong45.5%

27 doctors read X-rays with help from an AI that researchers set to give some wrong answers. When it was wrong, doctors at every experience level often went along with it.

Clue: Knowing a lot is not enough. Checking is what protects you.

Dratsch et al., Radiology, 2023

Good news: spotting mistakes can be learned

A little know-how goes a long way.

+26.5%

Better at telling true from false

A short set of tips on spotting false news made a nationally representative group of US adults 26.5% better at telling real headlines from fake ones, and it still showed weeks later. A highly educated online group in India improved by 17.5%, though a rural group with little social media use did not change.

Guess et al., PNAS, 2020

Honest AI helps

Showing doubt makes it easier to judge

When AI explanations were changed to match how sure the AI really was, people got better at telling its right answers from its wrong ones.

Steyvers et al., Nature Machine Intelligence, 2025

Detective levels, from 5 to 14

The picture of AI gets more detailed as kids grow. The habit of checking stays the same.

5–7

Level 1

A clever guessing machine

Young children can understand that AI makes guesses, and that guesses can be wrong even when they sound grown-up.

Try asking“Is that a fact, or is it guessing? How could we check?”

8–10

Level 2

Catch it out

Kids love testing AI with questions they already know the answer to. Finding a mistake themselves is the lesson that sticks.

Try asking“Ask it something you know really well. Did it get it right?”

11–14

Level 3

Know where it breaks

Older kids can learn where AI slips most — exact numbers, quotes, sources and recent news — and how to check them somewhere else.

Try asking“Where would you look to check that is actually true?”

Your detective checklist

Five quick questions for any AI answer. Tick them off as you go.

 

Do not be scared of AI. Just know when to check.

Trust everything and you pick up its mistakes. Trust nothing and you miss a useful tool. AI literacy is the middle: knowing how it works, where it slips, and feeling able to say so.

Sources

  1. 1.Parasuraman, R. & Riley, V. (1997). Humans and automation: Use, misuse, disuse, abuse. Human Factors, 39(2), 230–253. doi.org/10.1518/001872097778543886
  2. 2.Walters, W. H. & Wilder, E. I. (2023). Fabrication and errors in the bibliographic citations generated by ChatGPT. Scientific Reports, 13, 14045. doi.org/10.1038/s41598-023-41032-5
  3. 3.Steyvers, M. et al. (2025). What large language models know and what people think they know. Nature Machine Intelligence, 7, 221–231. doi.org/10.1038/s42256-024-00976-7
  4. 4.Dratsch, T. et al. (2023). Automation bias in mammography: The impact of artificial intelligence BI-RADS suggestions on reader performance. Radiology, 307(4), e222176. doi.org/10.1148/radiol.222176
  5. 5.Guess, A. M. et al. (2020). A digital media literacy intervention increases discernment between mainstream and false news in the United States and India. PNAS, 117(27), 15536–15545. doi.org/10.1073/pnas.1920498117