Issues

• MultiPL-E pass@k averages every *.results.json in the shared system temp dir, so leftovers from earlier runs or the previous task are scored too

  bigcode-project/bigcode-evaluation-harness#327 · Filter by author shaurya416 opened on Sep 23, 2026

• Verify evals on Papers with Code

  bigcode-project/bigcode-evaluation-harness#326 · Filter by author NielsRogge opened on Sep 14, 2026

• Optional EvalPort export for task datasets and pass@k results

  bigcode-project/bigcode-evaluation-harness#325 · Filter by author adhabnr-ux opened on Aug 22, 2026

• 📝 Integration Proposal: CAJAL — Scientific Paper Agent for BigCode

  bigcode-project/bigcode-evaluation-harness#320 · Filter by author Agnuxo1 opened on May 4, 2026

• Wrong Hugging Face link for Spider dataset in bigcode-evaluation-harness/docs/README.md line 434

  bigcode-project/bigcode-evaluation-harness#315 · Filter by author 354246695 opened on Sep 9, 2025
  1

• Support configurability of FIM tokens on SantaCoder

  bigcode-project/bigcode-evaluation-harness#313 · Filter by author Jay-Roberts opened on Jul 10, 2025

• testing Humaneval of qwen-2.5-7B-coder-instruct

  bigcode-project/bigcode-evaluation-harness#308 · Filter by author zxiangx opened on Mar 26, 2025

• how to add new model?

  bigcode-project/bigcode-evaluation-harness#307 · Filter by author pengzhangzhi opened on Mar 26, 2025

• When executing languages such as JS and GO in Multiple-E, the generated results suddenly end

  bigcode-project/bigcode-evaluation-harness#306 · Filter by author fxnie opened on Mar 25, 2025

• Unable to execute the MultiPL-E task for C++

  bigcode-project/bigcode-evaluation-harness#304 · Filter by author JohnneyQin opened on Mar 11, 2025
  1

• Code-Llama-7B-Python 4 Bit Error on HumanEval

  bigcode-project/bigcode-evaluation-harness#303 · Filter by author wilyub opened on Feb 25, 2025

• is there a benchmark page on the benchmark results evaluated using bigcode-evaluation-harness

  bigcode-project/bigcode-evaluation-harness#300 · Filter by author yxchng opened on Jan 21, 2025
  1

• Select device: GPU for model eval

  bigcode-project/bigcode-evaluation-harness#297 · Filter by author kn0wn-cyber opened on Dec 19, 2024

• HumanEval-X generation appears to not time out called subproceses

  bigcode-project/bigcode-evaluation-harness#290 · Filter by author nielstron opened on Nov 24, 2024

• Adding an assistant_model Argument for Speculative Decoding

  bigcode-project/bigcode-evaluation-harness#288 · Filter by author ilyasoulk opened on Nov 23, 2024

• Could you share a completed file of generations_mbppplus.json

  bigcode-project/bigcode-evaluation-harness#287 · Filter by author FearandDreams1123 opened on Nov 13, 2024

• mbpp pass@1 is lower than mbppplus pass@1 on starcoder2?

  bigcode-project/bigcode-evaluation-harness#285 · Filter by author Liuzzyg opened on Nov 6, 2024

• Continuing / Extending Previous Results from Generating and Evaluating?

  bigcode-project/bigcode-evaluation-harness#284 · Filter by author RylanSchaeffer opened on Nov 1, 2024

• Docker image for multiple evalulation broken

  bigcode-project/bigcode-evaluation-harness#283 · Filter by author Extirpater opened on Oct 27, 2024
  1

• [REQUEST] Apply tokenizer chat template for HumanEvalPack

  bigcode-project/bigcode-evaluation-harness#282 · Filter by author timrbula opened on Oct 25, 2024
  1

• [REQUEST] support model-parallel evaluation for big models

  bigcode-project/bigcode-evaluation-harness#278 · Filter by author tarahjjeon opened on Oct 18, 2024

• trust_remote_code is not been passed to dataset

  bigcode-project/bigcode-evaluation-harness#276 · Filter by author login256 opened on Sep 24, 2024
  1

• ValueError: Infilling not yet supported for:/Meta-Llama-3.1-8B

  bigcode-project/bigcode-evaluation-harness#275 · Filter by author kbmlcoding opened on Sep 23, 2024

• Evaluation result of bigcode/starcoder2-3b on gsm8k_pal does not matched the paper

  bigcode-project/bigcode-evaluation-harness#272 · Filter by author nongfang55 opened on Sep 13, 2024

• Evaluating a Model with a Local Dataset in an Offline Environment

  bigcode-project/bigcode-evaluation-harness#271 · Filter by author ankush13r opened on Sep 12, 2024
  9