HLE
Canonical page on the main site: sinoaihub.com/benchmarks/hle
Type
benchmark
Key facts
- Task type: Expert-level academic Q&A across mathematics, humanities and natural sciences
- Dataset size: 2,500 questions across dozens of subjects
- Evaluation method: Multiple-choice and short-answer questions developed by subject-matter experts; multimodal; suitable for automated grading
- Scoring: Accuracy (% correct); frequently reported with and without tool access
- Recorded evaluations: 7
Description
Humanity's Last Exam - a frontier benchmark of expert-level questions across disciplines, often reported with and without tool access.
Evaluations
| benchmark | model | score | model_version | metric | date | source_type | source_url |
|---|---|---|---|---|---|---|---|
| HLE | deepseek-v4-1-flash | 36.8 | 39.1 on pure-text subset | accuracy | 2026-09-10 | vendor_reported | https://api-docs.deepseek.com/updates |
| HLE | deepseek-v4-pro | 42.7 (60.0 with tools) | — | accuracy | 2026-08-13 | vendor_reported | https://api-docs.deepseek.com/updates |
| HLE | qwen3.8-max | 43.6 (56.2 with tools) | — | accuracy | 2026-08 | vendor_reported | https://huggingface.co/Qwen/Qwen3.8-2.4T-A95B |
| HLE | kimi-k3 | 43.5 (56.0 with tools) | HLE-Full | accuracy | 2026-07 | vendor_reported | https://github.com/MoonshotAI/Kimi-K3 |
| HLE | kimi-k2.5 | 30.1 (50.2 with tools) | HLE-Full | accuracy | — | vendor_reported | https://github.com/MoonshotAI/Kimi-K2.5 |
| HLE | minimax-m2 | 12.5 without tools / 31.8 with tools | — | accuracy | 2025-10 | vendor_reported | https://github.com/MiniMax-AI/MiniMax-M2 |
Methodology
Task type: Expert-level academic Q&A across mathematics, humanities and natural sciences
Dataset size: 2,500 questions across dozens of subjects
Evaluation method: Multiple-choice and short-answer questions developed by subject-matter experts; multimodal; suitable for automated grading
Scoring: Accuracy (% correct); frequently reported with and without tool access
Contamination notes: Dataset includes a canary string (hle:3r2s:26b5c67b-...) to aid model builders in filtering the dataset from future training.
Relevant Models
- Qwen3.8-Max — 43.6 (56.2 with tools)
- DeepSeek-V4.1-Flash — 36.8
- Kimi K3 — 43.5 (56.0 with tools)
- DeepSeek-V4-Pro — 42.7 (60.0 with tools)
Limitations
All scores are vendor-reported and not independently verified. With-tools and without-tools results are not directly comparable; the setting is recorded per score.
Verification Status
verified
Benchmark Changes
No documented benchmark changes on record as of 2026-09-29.
Last Verified
2026-09-20
Source history
No documented source-change events located as of 2026-09-20.
Sources
| evidence_id | source_name | source_url | source_type | published | verified | confidence | conflict |
|---|---|---|---|---|---|---|---|
| src-benchmarks-hle-1 | DeepSeek API Change Log | https://api-docs.deepseek.com/updates | Official documentation | — | 2026-09-20 | high | — |
| src-benchmarks-hle-2 | Qwen3.8-2.4T-A95B model card | https://huggingface.co/Qwen/Qwen3.8-2.4T-A95B | Model card | — | 2026-09-20 | high | — |
| src-benchmarks-hle-3 | Kimi K3 GitHub README | https://github.com/MoonshotAI/Kimi-K3 | Official documentation | — | 2026-09-20 | high | — |