MMMU-Pro
Canonical page on the main site: sinoaihub.com/benchmarks/mmmu-pro
Type
benchmark
Key facts
- Task type: Multimodal understanding and reasoning (college-level, multi-discipline)
- Dataset size: 1,730 questions in standard format plus 1,730 vision-augmented variants (3,460 total); parent MMMU = 11.5K questions across 6 disciplines, 30 subjects
- Evaluation method: Multiple-choice questions with interleaved images; vision-only input setting removes text leakage
- Scoring: Accuracy (% correct answers)
- Recorded evaluations: 3
Description
Multimodal, multi-discipline understanding benchmark with college-level questions requiring reasoning.
Evaluations
| benchmark | model | score | metric | date | source_type | source_url |
|---|---|---|---|---|---|---|
| MMMU-Pro | kimi-k3 | 81.6 (83.4 with tools) | accuracy | 2026-07 | vendor_reported | https://github.com/MoonshotAI/Kimi-K3 |
| MMMU-Pro | kimi-k2.5 | 78.5 | accuracy | — | vendor_reported | https://github.com/MoonshotAI/Kimi-K2.5 |
Methodology
Task type: Multimodal understanding and reasoning (college-level, multi-discipline)
Dataset size: 1,730 questions in standard format plus 1,730 vision-augmented variants (3,460 total); parent MMMU = 11.5K questions across 6 disciplines, 30 subjects
Evaluation method: Multiple-choice questions with interleaved images; vision-only input setting removes text leakage
Scoring: Accuracy (% correct answers)
Relevant Models
- Kimi K3 — 81.6
Limitations
All scores are vendor-reported and not independently verified. With-tools and without-tools results are not directly comparable.
Verification Status
verified
Benchmark Changes
No documented benchmark changes on record as of 2026-09-29.
Last Verified
2026-09-20
Source history
No documented source-change events located as of 2026-09-20.
Sources
| evidence_id | source_name | source_url | source_type | published | verified | confidence | conflict |
|---|---|---|---|---|---|---|---|
| src-benchmarks-mmmu-pro-1 | Kimi K3 GitHub README | https://github.com/MoonshotAI/Kimi-K3 | Official documentation | — | 2026-09-20 | high | — |
| src-benchmarks-mmmu-pro-2 | Kimi K2.5 GitHub README | https://github.com/MoonshotAI/Kimi-K2.5 | Official documentation | — | 2026-09-20 | high | — |