René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Test Qwen2.5-Coder-32b-instruct pass@10","articleBody":"Hi,\n\nI’m currently testing Qwen2.5-Coder-32B-Instruct and trying to compute pass@10, but the evaluation stage seems to become nearly endless, particularly during the test execution phase.\n\nIn addition, I’m encountering OOM issues even on fairly large CPU clusters. From the behavior, it seems possible that worker processes are recursively creating more workers, which may be contributing to both the extremely long runtime and the memory blow-up.\n\nI’m wondering:\n\n* Is this a known issue with the evaluator?\n* Are there recommended settings for pass@10 evaluation to avoid runaway execution?\n* Should the number of workers be restricted manually, or is there a safer evaluation mode for large-scale runs?\n\nAny advice would be appreciated. Thanks.\n","author":{"url":"https://github.com/hyperplane2021-hub","@type":"Person","name":"hyperplane2021-hub"},"datePublished":"2026-03-19T19:50:25Z","url":"https://github.com/bigcode-project/bigcodebench/issues/120"}