Commit0

Properties

axis: coding
verification: det-test
whatItTests: Challenges an agent to write Python libraries from scratch given only an API specification document and interactive unit tests, with static-analysis and execution feedback available across multiple stages.
discriminates: [build-verify-reflect]
saturation: open
patternLabFit: The multi-stage static-analysis-plus-execution feedback loop is one of the purest build-verify-reflect setups available here: unlike bug-patch benchmarks there is no existing codebase to anchor the agent, so verification failure compounds across stages. The paper tests single agents only.
source: Commit0: Library Generation from Scratch (arXiv:2412.01769)
notes: Distinct from every proposed issue-resolution or completion benchmark here: there is no reference implementation to patch or complete, only an API spec and tests, testing generative software architecture rather than localized fixes. Library-count claim ('54 libraries') was reported by third-party summaries, not the fetched abstract, so it is omitted here rather than asserted.

axis: coding
verification: det-test
whatItTests: Challenges an agent to write Python libraries from scratch given only an API specification document and interactive unit tests, with static-analysis and execution feedback available across multiple stages.
discriminates: [build-verify-reflect]
saturation: open
patternLabFit: The multi-stage static-analysis-plus-execution feedback loop is one of the purest build-verify-reflect setups available here: unlike bug-patch benchmarks there is no existing codebase to anchor the agent, so verification failure compounds across stages. The paper tests single agents only.
source: Commit0: Library Generation from Scratch (arXiv:2412.01769)
notes: Distinct from every proposed issue-resolution or completion benchmark here: there is no reference implementation to patch or complete, only an API spec and tests, testing generative software architecture rather than localized fixes. Library-count claim ('54 libraries') was reported by third-party summaries, not the fetched abstract, so it is omitted here rather than asserted.