SWE-bench Multimodal

Properties

axis: coding
verification: det-test
whatItTests: 617 GitHub issue-fixing tasks drawn from 17 JavaScript UI/visualization libraries where the problem statement or unit tests include images, testing whether a coding agent can localize and fix visual, user-facing bugs.
discriminates: [build-verify-reflect]
saturation: open
patternLabFit: Uses the same test-harness-as-oracle build-verify-reflect structure as SWE-bench Verified/Pro but adds an image-grounding requirement absent from either tracked entry. The paper compares existing single-agent systems (SWE-agent, Agentless, etc.), not multi-agent patterns.
source: SWE-bench Multimodal: Do AI Systems Generalize to Visual Software Domains? (arXiv:2410.03859)
notes: Distinct from swe-bench-verified and swe-bench-pro (both Python-only, text-only issues): this variant is JavaScript, front-end/visual, and requires multimodal (image) grounding to localize and verify fixes, a capability axis the tracked SWE-bench entries do not test.

axis: coding
verification: det-test
whatItTests: 617 GitHub issue-fixing tasks drawn from 17 JavaScript UI/visualization libraries where the problem statement or unit tests include images, testing whether a coding agent can localize and fix visual, user-facing bugs.
discriminates: [build-verify-reflect]
saturation: open
patternLabFit: Uses the same test-harness-as-oracle build-verify-reflect structure as SWE-bench Verified/Pro but adds an image-grounding requirement absent from either tracked entry. The paper compares existing single-agent systems (SWE-agent, Agentless, etc.), not multi-agent patterns.
source: SWE-bench Multimodal: Do AI Systems Generalize to Visual Software Domains? (arXiv:2410.03859)
notes: Distinct from swe-bench-verified and swe-bench-pro (both Python-only, text-only issues): this variant is JavaScript, front-end/visual, and requires multimodal (image) grounding to localize and verify fixes, a capability axis the tracked SWE-bench entries do not test.