nuprl / MultiPL-E

A multi-programming language benchmark for evaluating the performance of large language model of code.

https://nuprl.github.io/MultiPL-E/

nuprl/MultiPL-E Issues

How to run a multi-GPU model for inference testing?
Updated a month ago
Citation for the LeetCode Dataset
Updated 2 months ago1
Quantized model is not supported - Calling cuda() is not supported for 4-bit or 8-bit quantized models
Closed 3 months ago2
leetcode dataset not found
Updated 3 months ago1
The "automodel" file does not have this parameter.
Closed 5 months ago
code generated with wrong end of string place
Closed 5 months ago2
padding left some token causing compile error
Closed 5 months ago2
Unable to load weights from pytorch checkpoint file
Closed 5 months ago1
Evaluation with a container stops halfway without error message
Closed 6 months ago4
All non-multiline commented prompts currently broken
Closed 8 months ago
R prompts are currently broken
Closed 8 months ago
Turn translator into a library
Closed 9 months ago1
Could I get all statistics?
Closed 10 months ago2
load_dataset() doesn't work without specifying dataset_revision?
Updated 10 months ago1
Error evaluating TS/Java
Updated a year ago3
Strange Scala Unit test Translation for output Tuple Type w/ extra parenthesis
Updated a year ago4
Add HumanEval+ tests
Closed a year ago15
Scala tests comparing optional value
Closed a year ago3
Small issues with Swift prompt signatures
Closed a year ago3
C# test sequence equality
Updated a year ago1
PHP test indexed array comparison
Updated a year ago
Perl Unit test comparing float values
Closed a year ago4
Perl Unit test when expecting "False/0" output
Updated a year ago5
Environment for evaluating C#
Closed a year ago13
Racket unit test numerical equivalence
Closed a year ago4
R unit test comparison between integer and double
Closed a year ago8
R unit tests atomic vector comparison
Closed a year ago7
C++ test float comparison
Closed a year ago3
Java transpiled test failing with optional output
Closed a year ago2
Stop tokens for Java do not allow completions that produce several top-level methods.
Closed a year ago1
Reported pass@k silently wrong for n<k
Closed a year ago1
Warning: Bash performance results artificially low
Closed a year ago5
Support execution for the FIM benchmarks
Closed a year ago1
FYI, HumanEval 95 check dict case canonical solution is wrong
Closed a year ago1
Java program evaluation error with javatuples.Pair class
Closed a year ago2
Fix some easy paths to wrong evaluation results
Closed a year ago5
Doesn't check for cuda support before attempting to execute on GPU.
Closed a year ago1
Better isolation for evaluation
Closed a year ago1
merge code from cd58e0314c9fefe8095149f07f9004248d1dbf94
Closed a year ago
Bad filenames when using gz
Closed 2 years ago