princeton-nlp / SWE-bench

[ICLR 2024] SWE-Bench: Can Language Models Resolve Real-world Github Issues?

https://www.swebench.com

princeton-nlp/SWE-bench Issues

what's the difference between environment_setup_commit and base_commit?
Updated 18 days ago
how to download one task instance from SWE-bench dataset?
Updated 18 days ago
What's the best way to browse the SWE-bench dataset?
Updated 18 days ago
How can one participate in the SWE-bench leaderboard?
Closed 20 days ago3
Dockerization of run_evaluation.py
Updated 21 days ago5
Has anyone successfully ran an eval on patches against early versions of astropy, sympy, scipy etc? I'm really struggling to run things from earlier python versions
Updated 22 days ago2
Issue with Django gold patch results not evaluating to "Resolved" when logs say "All tests passed"
Updated 22 days ago7
Error when running evaluation script - ModuleNotFoundError: No module named 'swebench'
Closed 2 months ago4
Using `uv pip` instead of `pip` for significant speedup
Updated 23 days ago1
swe-bench eval stops running after a point
Updated 25 days ago2
swe-bench eval stops running after a point
Closed a month ago1
run_live.py: clone_repo() takes 3 positional arguments but 5 were given
Updated a month ago
Is it possible to evaluate the train set?
Updated a month ago
Why AutoCodeRover not mentioned?
Updated a month ago1
How can I run_evalution? What is the args swe_bench_tasks,I need to create by myself?
Updated a month ago3
Reproducer Docker image
Updated a month ago
get_eval_refs doesn't work with a dataset that's been `save_to_disk`'d
Closed a month ago2
environment is lost when running pip install
Updated a month ago
Error when running evaluation - ModuleNotFoundError: No module named 'conda'
Updated a month ago11
conda activate && conda install gxx_linux-64 gcc_linux-64 make -y' returned non-zero exit status 2 and Syntax error: "(" unexpected
Closed a month ago8
improve eval performance by caching per-repo/version conda environments
Updated a month ago
`/bin/sh: pytest: command not found` when running evaluations
Updated 2 months ago2
Repository not found while running python3 create_text_dataset.py
Updated 2 months ago1
Dataset field & set up reliable environment
Updated 2 months ago
Issues in the test case parsing logic from the logs
Updated 2 months ago4
Unable to run evaluation. Testbed is having trouble creating a specified conda environment
Updated 2 months ago15
What are expected to submit for the leaderboard integration?
Closed 2 months ago4
Remove `pre_install` from install specs
Updated 2 months ago
Adding LLM tokens to the generated inference for cost calculation
Updated 2 months ago
Failed to install low versions of Python with Conda
Updated 2 months ago1
Unable to replicate basic results
Updated 2 months ago2
Sharing my gpt-4-bm25-27k results
Closed 2 months ago1
How to run evaluation with swe-bench_lite?
Closed 2 months ago5
Upper bound score by skilled human?
Closed 2 months ago2
PASS_TO_PASS tests failing on original program
Updated 2 months ago1
Unable to test for Scikit-Learn
Updated 2 months ago2
Request for Baseline Model Testing Log Files for Research Purposes
Closed 2 months ago1
Running SWE-Llama(13B/7B) on HuggingFace Endpoints [FeatureRequest]
Closed 2 months ago1
Reproducible Experiments via MLFlow or Similar [FeatureRequest]
Closed 2 months ago1
inference issue run live
Updated 2 months ago1
Difficulty Reproducing BM25 Results
Updated 2 months ago1
Running create_text_dataset.py gets Killed and takes too long
Updated 2 months ago2
lv3-0
Closed 2 months ago
UnboundLocalError: local variable 'env_name' referenced before assignment in harness/utils.py line 56
Closed 2 months ago1
Updated Docker Container
Closed 3 months ago3
Clarification Needed on Evaluation Code for Retrieval with BM25 and Context Length-Specific Recall Metrics
Closed 3 months ago3
Unable to create a mirror of an repository
Closed 3 months ago2
some wrong in build_dataset.py
Closed 3 months ago2
Argument `--swe_bench_tasks` is unclear
Closed 5 months ago4
Unreliability when generating patches with `diff` format?
Closed 5 months ago2