AutomationBench
Canonical page on the main site: sinoaihub.com/benchmarks/automationbench
Type
benchmark
Key facts
- Task type: End-to-end business workflow automation in simulated SaaS environments
- Dataset size: 47 simulated SaaS tools across 6 business functions (Sales, Marketing, Operations, Support, Finance, HR)
- Evaluation method: Each task initializes a simulated business environment (CRM, calendar, inbox); the agent must leave the environment in the correct end state
- Scoring: Task success rate (environment end-state verification)
- Recorded evaluations: 5
Description
Benchmark of computer-use automation tasks.
Evaluations
| benchmark | model | score | metric | date | source_type | source_url | model_version |
|---|---|---|---|---|---|---|---|
| AutomationBench | deepseek-v4-1-flash | 54.8 | accuracy | 2026-09-10 | vendor_reported | https://api-docs.deepseek.com/updates | — |
| AutomationBench | deepseek-v4-pro | 31.8 | accuracy | 2026-08-13 | vendor_reported | https://api-docs.deepseek.com/updates | Public |
| AutomationBench | qwen3.8-max | 27.3 | accuracy | 2026-08 | vendor_reported | https://huggingface.co/Qwen/Qwen3.8-2.4T-A95B | Pass@1 |
| AutomationBench | glm-5.3-flash | 48.8 | accuracy | 2026-08-26 | vendor_reported | https://docs.z.ai/guides/vlm/glm-5.3-flash | GLM-5.2: 26.2 |
Methodology
Task type: End-to-end business workflow automation in simulated SaaS environments
Dataset size: 47 simulated SaaS tools across 6 business functions (Sales, Marketing, Operations, Support, Finance, HR)
Evaluation method: Each task initializes a simulated business environment (CRM, calendar, inbox); the agent must leave the environment in the correct end state
Scoring: Task success rate (environment end-state verification)
Relevant Models
- GLM-5.3-Flash — 48.8
Limitations
All scores are vendor-reported and not independently verified. Pass@1 vs other sampling settings differ between vendors.
Verification Status
verified
Benchmark Changes
No documented benchmark changes on record as of 2026-09-29.
Last Verified
2026-09-20
Source history
No documented source-change events located as of 2026-09-20.
Sources
| evidence_id | source_name | source_url | source_type | published | verified | confidence | conflict |
|---|---|---|---|---|---|---|---|
| src-benchmarks-automationbench-1 | DeepSeek API Change Log | https://api-docs.deepseek.com/updates | Official documentation | — | 2026-09-20 | high | — |
| src-benchmarks-automationbench-2 | Z.ai docs — GLM-5.3-Flash model page | https://docs.z.ai/guides/vlm/glm-5.3-flash | Official documentation | — | 2026-09-20 | high | — |