OV OmegaUse-OfficeVal

OmegaUse-OfficeVal: Benchmarking LLM Agents on Long-Horizon Office-Suite Tasks with Economic Grounding

Jingbo Zhou*†, Yusai Zhao*†, Qi Bao, Jingjia Cao, Zhenghai Chen, Chang Gao, Kaiqi Guo, Muxin Guo, Mingxuan Li, Xinjiang Lu, Yanru Ma, Yixiong Xiao, Zenghui Zhang, Le Zhang, and Hua Wu†

Agent Frontier Team, Large Model Frontier Research Department, Baidu Inc.

Paper Code Data HF Paper Leaderboard Examples
100 tasks 220 input files 2.32 hours human labor time per task 20.09 task rubrics per task

Task examples

Task Examples of OmegaUse-OfficeVal

Six OmegaUse-OfficeVal task examples showing instructions, input files, task price proxy, human labor time, and golden office-suite artifacts.
Representative tasks preserve concise instructions while pairing them with input files, economically grounded annotations (human labor time and task price proxy), and a code verifier. View full-size figure

Abstract

A benchmark for evaluating LLM agents on economically grounded, long-horizon office-suite tasks.

Large language model (LLM) agents are increasingly expected to assist users in completing tasks. However, existing benchmarks provide limited support for evaluating whether agents can carry out office-suite workflows at a reasonable cost. We introduce OmegaUse-OfficeVal, a benchmark for evaluating LLM agents on long-horizon office-suite tasks with task-level economic grounding. The benchmark comprises 100 tasks derived from office-suite requests proposed by practitioners and adapted through a privacy-preserving process. On average, these tasks require 2.32 hours of human labor to complete. An important feature of the benchmark is that each task is paired with two economic signals: human labor time and task price proxy. These signals enable direct comparisons between human costs and LLM inference costs, as well as value-weighted evaluation. To support stable evaluation, we develop code-based verifiers from fine-grained rubrics. We evaluate several frontier LLMs together with a human baseline. Although all evaluated LLMs are substantially cheaper and faster than human workers, they have not yet approached human-level deliverable quality. The code and dataset are fully open-sourced, and more information is available on our project website: https://omegause-officeval.github.io/.

Task overview

100 authentic tasks reflecting long-horizon Office-Suite work.

100 Office-suite tasks
220 Input files
115 Output artifacts
2.32h Mean human labor time
2,009 Task rubrics

Files and economic signals

Multimodal inputs, editable deliverables, and task-level economically grounded annotation.

File type Input files Output artifacts
Video / Audio100
Image770
DOCX6348
PPTX3140
XLSX2524
PDF143
Total220115
Human labor time 2.32 hours mean

2.03 hours median, with the longest task requiring 8.35 hours.

Task price proxy $6.86 mean

$5.11 median, combining practitioner prices and three-expert estimates.

Evaluation target Final deliverables

Trajectory-agnostic scoring supports GUI, scripts, APIs, and hybrid workflows.

Histograms of task price proxy and human labor time across the 100 benchmark tasks.
Task price proxy ranges from $0.88 to $29.20, while human labor time ranges from 0.08 to 8.35 hours. View full-size figure

Task distribution

Sunburst chart of 100 OmegaUse-OfficeVal tasks by professional domain and operation intent.
The inner ring shows the domain distribution, and the outer ring shows the operation-intent distribution within each domain. View full-size figure

Construction pipeline

From 1,715 practitioner proposals to 100 verified long-horizon Office-suite tasks.

OmegaUse-OfficeVal construction pipeline: practitioner task collection and screening, economically grounded value estimation, and iterative code-based verification.
The benchmark combines authentic task collection, privacy-preserving artifact reconstruction, human labor and price annotation, and expert-aligned executable verification. View full-size figure
Task funnel 1,715 → 595 → 282 → 100

Practicality filtering and three-expert screening retain only authentic, feasible long-horizon work.

Economic grounding 20 qualified annotators

At least two people complete each task; the mean of the two shortest valid times defines human labor time.

Code-based Verification 2,009 task rubrics

Task rubrics are translated into executable checks and revised until human and code judgments align.

Leaderboard

Quality, time, and cost reveal different model strengths.

Human quality 27.79

Highest overall score, but still far from perfect on these demanding workflows.

Best LLM quality GLM-5.2 · 17.91

The strongest average deliverable score among the evaluated LLMs.

Best LLM value Qwen3.7-Plus

Highest time- and price-weighted LLM scores, at $0.2152 per task.

Fastest runtime DeepSeek-V4-Pro

0.184 hours per task on average.

Primary metric
Rank

Time- and price-weighted scores weight each task by its annotated human labor time or task price proxy. Human cost is the mean task price proxy; LLM cost is inference expenditure. Time is reported in hours per task.

Analysis

LLMs are much cheaper and faster, but the deliverable-quality gap remains.

Bubble chart comparing benchmark score, time per task, and cost per task for humans and five LLMs.
Humans achieve the best deliverable quality, while every evaluated LLM is substantially faster and cheaper. Bubble size represents average cost per task. View full-size figure
Stacked bars showing task score buckets for the human baseline and five LLMs.
Humans score above 50 on 21% of tasks. Qwen3.7-Plus has the lowest zero-score rate among LLMs at 38%. View full-size figure
Heatmap of average score by human labor-time bucket for humans and five LLMs.
Performance generally falls as human labor time increases, especially for LLMs on tasks exceeding three hours. View full-size figure
Explore domain, output-file, and operation-intent breakdowns
Heatmap of average task score by professional domain for humans and five LLMs.
Average task score by professional domain. Financial Data is particularly challenging for most LLMs. View full-size figure
Heatmap of average task score by output file type for humans and five LLMs.
Average task score by output file type. Mixed-output workflows remain especially difficult. View full-size figure
Heatmap of average task score by operation intent for humans and five LLMs.
Average task score by operation intent. Annotation and extraction produce the strongest LLM results. View full-size figure
Coding Actions vs. Computer-Use Actions on Office-Suite Tasks

Should an office agent work through visual interfaces, manipulate files with code, or combine both? Using K2.6, we compare three CoAct variants across OmegaUse-OfficeVal and OSWorld office-related tasks.

Bar chart comparing CoAct-Hybrid, CoAct-Coding-Only, and CoAct-CUA-Only on OmegaUse-OfficeVal. OmegaUse-OfficeVal Bar chart comparing CoAct-Hybrid, CoAct-Coding-Only, and CoAct-CUA-Only on OSWorld office-related tasks. OSWorld office-related tasks
Figure 8. CoAct variants with K2.6 on OmegaUse-OfficeVal and OSWorld office-related tasks. Scores should be compared within each benchmark because their tasks and evaluation protocols differ.

Long-horizon deliverables favor coding. On OmegaUse-OfficeVal, Coding-Only leads at 14.41, Hybrid follows at 13.77, and CUA-Only falls to 0.78.

Visual interaction still matters. On OSWorld office tasks, Hybrid leads at 77.58, ahead of CUA-Only at 68.90 and Coding-Only at 55.62.

Takeaway: there is no single best action mode for every office task. Coding provides reliable file-level execution for long workflows, while computer use remains valuable for visual grounding and application-level interaction. Learning when to route between the two remains an important open problem.

Resources

Paper, dataset, and verification code are openly available.

Paper arXiv:2607.27155 Read the technical report Code Evaluation framework Apache-2.0 licensed source Dataset 100 benchmark tasks Instructions, inputs, annotations, and evaluation assets Documentation Run the verifiers Package format, CLI usage, and reports

Reproduce

Evaluate a complete set of office-suite deliverables locally.

1

Prepare

Package task directories named officeval_001 through officeval_100 in a ZIP archive.

2

Evaluate

The CLI validates the archive and runs each verifier in an isolated subprocess.

3

Inspect

Review machine-readable JSON and CSV summaries together with per-task verifier reports.

Quick start

python -m pip install -e .
omegause-officeval --package /absolute/path/to/submission.zip

Platform note

Ninety-one verifiers run in normal cross-platform mode. Nine designated verifiers require Microsoft Office COM on Windows; in automatic mode they are skipped on macOS and Linux.

FAQ

Common equations and evaluation details.

Why attach economic signals to every task?

Human labor time and task price proxy enable human-versus-LLM cost comparisons and value-weighted evaluation.

Does the benchmark require a particular agent interface?

No. Evaluation targets final deliverables, so agents may use GUI actions, scripts, APIs, or hybrid workflows.

How is scoring stabilized?

Fine-grained rubrics are converted into executable checks and revised through expert review and human-code discrepancy resolution.

Citation

Cite OmegaUse-OfficeVal.

@misc{zhou2026omegauseofficeval,
  title        = {OmegaUse-OfficeVal: Benchmarking LLM Agents on Long-Horizon Office-Suite Tasks with Economic Grounding},
  author       = {Jingbo Zhou and Yusai Zhao and Qi Bao and Jingjia Cao and Zhenghai Chen and Chang Gao and Kaiqi Guo and Muxin Guo and Mingxuan Li and Xinjiang Lu and Yanru Ma and Yixiong Xiao and Zenghui Zhang and Le Zhang and Hua Wu},
  year         = {2026},
  eprint       = {2607.27155},
  archivePrefix = {arXiv},
  primaryClass = {cs.AI},
  url          = {https://arxiv.org/abs/2607.27155}
}

Team

Jingbo Zhou*†, Yusai Zhao*†, Qi Bao, Jingjia Cao, Zhenghai Chen, Chang Gao, Kaiqi Guo, Muxin Guo, Mingxuan Li, Xinjiang Lu, Yanru Ma, Yixiong Xiao, Zenghui Zhang, Le Zhang, and Hua Wu†

Agent Frontier Team, Large Model Frontier Research Department, Baidu Inc., China.

* Equal contribution. † Project co-lead and corresponding authors.

zhoujingbo@baidu.com zhaoyusai@baidu.com wu_hua@baidu.com