BullshitBench – Which AI Detects Nonsense?

In 30 Seconds

AI models keep getting better at writing, coding, and analyzing. But how good are they at detecting nonsense? The open-source benchmark BullshitBench by Peter Gostev asks exactly this question – and the answers are sobering.

The result: Over 45% of all tested nonsense questions are simply accepted by AI models on average. Only the best models reliably recognize when a question doesn't make sense.

What Is BullshitBench?

BullshitBench is a benchmark that confronts AI models with 100 intentionally nonsensical questions that sound plausible. The questions cover five domains:

• Software (40 questions)
• Finance (15 questions)
• Legal (15 questions)
• Medical (15 questions)
• Physics (15 questions)

Each question uses one of 13 nonsense techniques – for example:

Evaluation is done by a 3-judge panel of Claude Sonnet 4.6, GPT-5.2, and Gemini 3.1 Pro – three top models that rate responses into three categories.

The Three Rating Categories

• 🟢 Clear Pushback: The model clearly recognizes and rejects the nonsense
• 🟡 Partial Challenge: The model notices issues but still engages with the false premise
• 🔴 Accepted Nonsense: The model treats the nonsense as a valid question

The Results: Who Detects Bullshit?

Top 10 – The Best Nonsense Detectors

And the Others?

Results for GPT and Gemini models are significantly weaker:

The pattern is clear: Anthropic's Claude models dominate the top spots by a wide margin. The top 7 are exclusively Claude models. Only at rank 8 does Qwen 3.5 appear as the first non-Anthropic model.

Why Does This Matter?

1. Hallucination Isn't the Only Problem

The AI community talks a lot about hallucinations – when models invent facts. BullshitBench reveals a related but different problem: models that don't question false input and simply play along.

If you ask an AI a question based on a false assumption and it confidently answers – you have a bigger problem than a hallucination.

2. The "Yes-Man" Problem

Many models are trained to be helpful. This creates a bias: better to give an answer than none at all. BullshitBench shows which models overcome this reflex and instead say: "Wait, this question doesn't make sense."

3. Domain Differences Are Real

The differences between domains are fascinating:

• Physics: Most models detect nonsense best here (up to 100% for Claude Sonnet 4.6)
• Software: Middle of the pack – even good models fall for it more often
• Legal: Particularly difficult – plausible-sounding legal nonsense catches many models off guard

This means: Your AI's reliability depends heavily on the domain you're using it in.

What Does This Mean in Practice?

For Decision-Makers

If you use AI for business-critical decisions – contract analysis, financial planning, medical texts – the ability to detect nonsense is at least as important as the ability to give good answers.

Check whether your model can also say "No."

For Developers

If you build AI-powered workflows, think about validation steps. A model that accepts 60% of nonsense questions will also process faulty user inputs without pushback.

For AI Enthusiasts

BullshitBench is open source and available on GitHub. You can test your own models, contribute questions, or review the methodology.

The Meta Question: Is It Getting Better?

One of the most interesting visualizations in the BullshitBench viewer shows the trend over time: Are newer models getting better at detecting nonsense?

The answer is nuanced:

• Anthropic: Clear upward trend – each generation improves
• OpenAI: Little improvement between GPT-5 and GPT-5.4 in nonsense detection
• Google: Gemini shows progress in newer versions but remains behind Claude

This suggests that nonsense detection isn't an automatic byproduct of "bigger models" – it needs to be specifically trained.

Conclusion

BullshitBench is one of the most refreshing benchmarks in recent memory. Instead of measuring how well a model solves a task, it measures how well a model recognizes that it shouldn't solve the task at all.

For anyone using AI productively, this is a critical capability. Because the most dangerous scenario isn't an AI that says "I don't know" – it's one that confidently answers bullshit.

Three takeaways:

1. Anthropic's Claude dominates nonsense detection by a wide margin
2. Domain matters – test your model in your specific field
3. Nonsense detection is a standalone quality metric that's often missing from standard benchmarks

→ Open BullshitBench v2 Viewer → GitHub Repository