RepoBench

Properties

axis: coding
verification: det-gold
whatItTests: Repository-level code auto-completion evaluated across three linked tasks: retrieving relevant cross-file snippets (RepoBench-R), predicting the next line given cross-file and in-file context (RepoBench-C), and the combined retrieval-plus-completion pipeline (RepoBench-P), in Python and Java.
saturation: open
patternLabFit: The three-way retrieval/completion/pipeline split isolates whether failures come from retrieval (finding the right cross-file context) or generation (using it correctly), a decomposition that could motivate separating a retriever role from a generator role. The original paper evaluates single models only, so no collaboration-pattern discriminator is asserted.
source: RepoBench: Benchmarking Repository-Level Code Auto-Completion Systems (arXiv:2306.03091)
notes: Not new (2023, ICLR 2024) but explicitly named as in-scope and missing from the catalogue; distinct from BigCodeBench/SWE-bench family by testing repo-level auto-completion (predicting the next line/snippet from cross-file context) rather than issue resolution or full-function synthesis. Current-generation saturation state was not independently confirmed beyond the original paper; marked open pending validator check rather than asserting a specific figure.

axis: coding
verification: det-gold
whatItTests: Repository-level code auto-completion evaluated across three linked tasks: retrieving relevant cross-file snippets (RepoBench-R), predicting the next line given cross-file and in-file context (RepoBench-C), and the combined retrieval-plus-completion pipeline (RepoBench-P), in Python and Java.
saturation: open
patternLabFit: The three-way retrieval/completion/pipeline split isolates whether failures come from retrieval (finding the right cross-file context) or generation (using it correctly), a decomposition that could motivate separating a retriever role from a generator role. The original paper evaluates single models only, so no collaboration-pattern discriminator is asserted.
source: RepoBench: Benchmarking Repository-Level Code Auto-Completion Systems (arXiv:2306.03091)
notes: Not new (2023, ICLR 2024) but explicitly named as in-scope and missing from the catalogue; distinct from BigCodeBench/SWE-bench family by testing repo-level auto-completion (predicting the next line/snippet from cross-file context) rather than issue resolution or full-function synthesis. Current-generation saturation state was not independently confirmed beyond the original paper; marked open pending validator check rather than asserting a specific figure.