DevicesWorld

Properties

axis: agentic
verification: det-test
whatItTests: Cross-device collaborative task completion spanning mobile, desktop, and IoT device classes (6,140 tasks), where an agent must acquire information on one device and act on another; scored by rule-based verifiers that automatically check final device states and generated files against per-task success conditions.
discriminates: [build-verify-reflect]
saturation: open
patternLabFit: The rule-based, per-task verifier checking device state and generated files is directly analogous to OSWorld's execution-based checker, giving a build-verify-reflect hook; but the paper evaluates five single-agent frontier systems operating across devices, not a division-of-labor multi-agent setup, so no collaboration pattern beyond that is evidenced.
source: DevicesWorld: Benchmarking Cross-Device Agents in Heterogeneous Environments (arXiv:2607.13465)
notes: Distinct from OSWorld (single OS: Ubuntu/Windows/macOS within one environment) by requiring cross-device task completion; distinct from tau-bench (single-domain dialogue) and PlanBench-XL (single tool ecosystem). 'Collaborative operation' in the title refers to cross-device coordination by one agent, not multi-agent collaboration -- best system reached only 12.5% success, per the abstract.

axis: agentic
verification: det-test
whatItTests: Cross-device collaborative task completion spanning mobile, desktop, and IoT device classes (6,140 tasks), where an agent must acquire information on one device and act on another; scored by rule-based verifiers that automatically check final device states and generated files against per-task success conditions.
discriminates: [build-verify-reflect]
saturation: open
patternLabFit: The rule-based, per-task verifier checking device state and generated files is directly analogous to OSWorld's execution-based checker, giving a build-verify-reflect hook; but the paper evaluates five single-agent frontier systems operating across devices, not a division-of-labor multi-agent setup, so no collaboration pattern beyond that is evidenced.
source: DevicesWorld: Benchmarking Cross-Device Agents in Heterogeneous Environments (arXiv:2607.13465)
notes: Distinct from OSWorld (single OS: Ubuntu/Windows/macOS within one environment) by requiring cross-device task completion; distinct from tau-bench (single-domain dialogue) and PlanBench-XL (single tool ecosystem). 'Collaborative operation' in the title refers to cross-device coordination by one agent, not multi-agent collaboration -- best system reached only 12.5% success, per the abstract.