🗂️Benchmark Name: ODEX 📚Publisher: arxiv 🏠Author Affiliation: Carnegie Mellon University 🔗URL: https://github.com/zorazrw/odex/tree/master/data Number of Instances: 945 Problem Descrip...
Vgen
🗂️Benchmark Name: Vgen 📚Publisher: arxiv 🏠Author Affiliation: New York University 🔗URL: https://github.com/shailja-thakur/VGen Number of Instances: 17 Problem Description’s Natural Lan...
ERNIE-Code
📙Paper: ERNIE-Code Beyond English-Centric Cross-lingual Pretraining for Programming Languages 📚Publisher: arxiv 🏠Author Affiliation: Baidu 🔑Public: ✅ (promise) 🌐Architecture E...
DS-1000
🗂️Benchmark Name: DS-1000 📚Publisher: Arxiv 🏠Author Affiliation: The University of Hong Kong 🔗URL: https://ds1000-code-gen.github.io/ Number of Instances: 1,000 Problem Description’s N...
SecurityEval
🗂️Benchmark Name: SecurityEval 📚Publisher: ACM MSR4P&S 🏠Author Affiliation: University of Notre Dame 🔗URL: https://github.com/s2e-lab/SecurityEval Number of Instances: 130 Problem ...
BLOOM
📙Paper: BLOOM A 176B-Parameter Open-Access Multilingual Language Model 📚Publisher: arxiv 🏠Author Affiliation: BigScience 🔑Public: ✅ 🌐Architecture Encoder-Decoder Decoder...
TorchDataEval
🗂️Benchmark Name: TorchDataEval 📚Publisher: EMNLP 🏠Author Affiliation: Microsoft 🔗URL: https://github.com/microsoft/PyCodeGPT/tree/main/apicoder/private-eval/data Number of Instances: 50...
MBXP
🗂️Benchmark Name: MBXP 📚Publisher: Arxiv 🏠Author Affiliation: AWS AI Labs 🔗URL: https://github.com/amazon-research/mbxp-exec-eval Number of Instances: 974 per programming language Prob...
MBXP-MathQA
🗂️Benchmark Name: MBXP-MathQA 📚Publisher: Arxiv 🏠Author Affiliation: AWS AI Labs 🔗URL: not yet release Number of Instances: / Problem Description’s Natural Language: / Code Solution’...
MBXP-HumanEval
🗂️Benchmark Name: MBXP-HumanEval 📚Publisher: Arxiv 🏠Author Affiliation: AWS AI Labs 🔗URL: https://github.com/amazon-research/mbxp-exec-eval Number of Instances: 164 per programming langu...
Recently Updated