Keywords
repository localization
reference patches
exploratory regression
benchmark scope
graph retrieval
Abstract
Reference patches are convenient labels for repository localization, but their construction can restrict the retrieval question a benchmark can answer. This secondary analysis examines 300 public SWE-bench Lite test records from 12 repositories. Every reference patch changes one file, so file-count variation cannot support the initially considered association with issue-description length. We retain that finding and conduct an explicitly exploratory analysis of patch extent, defined as added plus deleted lines. The median patch changes six lines, and the median issue contains 150 word-like tokens. An ordinary least squares model of log one plus edited lines on log-base-two issue length, with repository fixed effects, estimates a slope of -0.0411 and an HC1 95 percent interval from -0.1295 to 0.0473. This weak descriptive association does not estimate retrieval effectiveness. The article explains why a single changed-file label cannot represent every file useful for understanding a task and proposes richer evaluation records for lexically anchored graph retrieval.
References
Hu, Yuntong, et al. "LARGER: Lexically Anchored Repository Graph Exploration and Retrieval." arXiv preprint arXiv:2605.16352 (2026).
Jimenez, Carlos E., et al. "SWE-bench: Can Language Models Resolve Real-World GitHub Issues?" International Conference on Learning Representations, 2024. https://arxiv.org/abs/2310.06770.
Princeton NLP. SWE-bench Lite. Hugging Face Datasets, public test split, JSON snapshot downloaded 25 Oct. 2026. https://huggingface.co/datasets/princeton-nlp/SWE-bench_Lite.
White, Halbert. “A Heteroskedasticity-Consistent Covariance Matrix Estimator and a Direct Test for Heteroskedasticity.” Econometrica, vol. 48, no. 4, 1980, pp. 817–838. https://doi.org/10.2307/1912934.
Cameron, A. Colin, and Douglas L. Miller. “A Practitioner’s Guide to Cluster-Robust Inference.” Journal of Human Resources, vol. 50, no. 2, 2015, pp. 317–372. https://doi.org/10.3368/jhr.50.2.317.
Roberts, David R., et al. “Cross-validation strategies for data with temporal, spatial, hierarchical, or phylogenetic structure.” Ecography, vol. 40, no. 8, 2017, pp. 913-929. https://doi.org/10.1111/ecog.02881.
Varma, Sudhir and Richard Simon. “Bias in error estimation when using cross-validation for model selection.” BMC Bioinformatics, vol. 7, no. 1, 2006. https://doi.org/10.1186/1471-2105-7-91.
Robertson, Stephen, and Hugo Zaragoza. "The Probabilistic Relevance Framework: BM25 and Beyond." Foundations and Trends in Information Retrieval, vol. 3, no. 4, 2009, pp. 333-389. https://doi.org/10.1561/1500000019.
Greenland, Sander, Judea Pearl, and James M. Robins. “Causal Diagrams for Epidemiologic Research.” Epidemiology, vol. 10, no. 1, 1999, pp. 37-48. https://doi.org/10.1097/00001648-199901000-00008.
Wilkinson, Mark D., et al. "The FAIR Guiding Principles for Scientific Data Management and Stewardship." Scientific Data, vol. 3, 2016, article 160018. https://doi.org/10.1038/sdata.2016.18.