VersiCode

• 📙Paper: VersiCode: Towards Version-controllable Code Generation
• 📚Publisher: arxiv
• 🏠Author Affiliation: Monash University, Nanjing University of Posts and Telecommunications, ByteDance Ltd., CSIRO’s Data61
• GitHub: https://github.com/wutong8023/VersiCode
• Contributions:
  • We propose two novel and important yet under-explored tasks in code generation, namely version-specific code completion (VSCC) and version-aware code editing (VACE);
  • We curate VersiCode, a new, large-scale version-controllable code generation dataset, aimed at comprehensively evaluating LLMs’ performance on the above two tasks;
  • We conduct comprehensive experiments on VersiCode, demonstrating that it is a high-quality and challenging dataset. The benchmarking results reveal that most LLMs struggle with version-specific code generation, especially with the latest libraries, highlighting areas for future exploration.