Papers
arxiv:2506.04535
BSBench: will your LLM find the largest prime number?
Published on Jun 5, 2025
Authors:
Abstract
A benchmark for evaluating LLMs on questions without reasonable answers reveals suboptimal performance of existing models.
Generated by Qwen/Qwen2.5-Coder-32B-Instruct
We propose that benchmarking LLMs on questions which have no reasonable answer actually isn't as silly as it sounds. We also present a benchmark that allows such testing and a method to modify the existing datasets, and discover that existing models demonstrate a performance far from the perfect on such questions. Our code and data artifacts are available at https://github.com/L3G5/impossible-bench
Get this paper in your agent:
hf papers read 2506.04535 Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash Models citing this paper 0
No model linking this paper
Cite arxiv.org/abs/2506.04535 in a model README.md to link it from this page.
Datasets citing this paper 0
No dataset linking this paper
Cite arxiv.org/abs/2506.04535 in a dataset README.md to link it from this page.
Spaces citing this paper 0
No Space linking this paper
Cite arxiv.org/abs/2506.04535 in a Space README.md to link it from this page.
Collections including this paper 0
No Collection including this paper
Add this paper to a collection to link it from this page.