DeepSWE
Canonical page on the main site: sinoaihub.com/benchmarks/deepswe
Type
benchmark
Key facts
- Task type: Long-horizon software engineering tasks on active open-source repositories
- Dataset size: 113 tasks across TypeScript, Go, Python, JavaScript and Rust
- Evaluation method: Isolated agent environment with program-based verifiers; the agent's committed patch is applied and graded in a pristine container
- Scoring: Binary reward plus pass fractions per task (reward.json / CTRF test report)
- Recorded evaluations: 7
Description
Software engineering benchmark built from real-world issues and pull requests.
Evaluations
| benchmark | model | score | model_version | metric | date | source_type | source_url |
|---|---|---|---|---|---|---|---|
| DeepSWE | deepseek-v4-1-flash | 74.2 | v1.1 | accuracy | 2026-09-10 | vendor_reported | https://api-docs.deepseek.com/updates |
| DeepSWE | deepseek-v4-pro | 62.7 | version not stated in source | accuracy | 2026-08-13 | vendor_reported | https://api-docs.deepseek.com/updates |
| DeepSWE | qwen3.8-max | 56.6 | v1.1 | accuracy | 2026-08 | vendor_reported | https://huggingface.co/Qwen/Qwen3.8-2.4T-A95B |
| DeepSWE | kimi-k3 | 67.5 | 67.3 with mini-SWE-agent harness | accuracy | 2026-07 | vendor_reported | https://github.com/MoonshotAI/Kimi-K3 |
| DeepSWE | glm-5.3 | 66.9 | v1.1 | accuracy | 2026-08-18 | vendor_reported | https://docs.z.ai/guides/llm/glm-5.3 |
| DeepSWE | glm-5.3-flash | 63.4 | v1.1 | accuracy | 2026-08-26 | vendor_reported | https://docs.z.ai/guides/vlm/glm-5.3-flash |
Methodology
Task type: Long-horizon software engineering tasks on active open-source repositories
Dataset size: 113 tasks across TypeScript, Go, Python, JavaScript and Rust
Evaluation method: Isolated agent environment with program-based verifiers; the agent's committed patch is applied and graded in a pristine container
Scoring: Binary reward plus pass fractions per task (reward.json / CTRF test report)
Relevant Models
- GLM-5.3 — 66.9
- GLM-5.3-Flash — 63.4
- DeepSeek-V4.1-Flash — 74.2
- Kimi K3 — 67.5
- DeepSeek-V4-Pro — 62.7
Limitations
All scores are vendor-reported and not independently verified. Some vendors do not state the benchmark version; unversioned scores should not be compared with versioned ones.
Verification Status
verified
Benchmark Changes
No documented benchmark changes on record as of 2026-09-29.
Last Verified
2026-09-20
Source history
No documented source-change events located as of 2026-09-20.
Sources
| evidence_id | source_name | source_url | source_type | published | verified | confidence | conflict |
|---|---|---|---|---|---|---|---|
| src-benchmarks-deepswe-1 | DeepSeek API Change Log | https://api-docs.deepseek.com/updates | Official documentation | — | 2026-09-20 | high | — |
| src-benchmarks-deepswe-2 | Z.ai docs — GLM-5.3 model page | https://docs.z.ai/guides/llm/glm-5.3 | Official documentation | — | 2026-09-20 | high | — |
| src-benchmarks-deepswe-3 | Kimi K3 GitHub README | https://github.com/MoonshotAI/Kimi-K3 | Official documentation | — | 2026-09-20 | high | — |