René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Qwen model generation degenerates into repetitive special tokens","articleBody":"Hi ! Nice Project!\nWhen I run the example_pythin.ipynb with Qwen2_5 7B Base as follows:\n```python\nimport sys\nsys.path.append('..')\nimport os\nfrom dotenv import load_dotenv  # 导入库\n\n# 1. 加载 .env 文件 (默认读取当前目录下的 .env)\nload_dotenv()\n# Assuming we are in the root directory\n\nfrom syncode import Syncode\nimport warnings\nwarnings.filterwarnings('ignore')\n\nmodel_name = os.getenv('QWEN2_5_7B_PATH')\n\n# Load the unconstrained original model\nllm = Syncode(model = model_name, mode='original', max_new_tokens=200)\n\n# Load the Syncode augmented model\nsyn_llm = Syncode(\n    model = model_name, \n    mode='grammar_mask', \n    grammar='python', \n    parse_output_only=False,\n    indent=True,\n    opp=False\n    )\n```\n\nStandard LLM generation is like \n```python\npartial_code = \"def is_prime(n):\\n    '''Return if prime'''\\n  \"\noutput = partial_code+llm.infer(partial_code)[0]\nprint(output)\n```\n\nwith output \n```\nSetting `pad_token_id` to `eos_token_id`:151643 for open-end generation.\ndef is_prime(n):\n    '''Return if prime'''\n  \n    if n == 1:\n        return False\n    if n == 2:\n        return True\n    if n \u003e 2 and n % 2 == 0:\n        return False\n    max_divisor = int(n**0.5) + 1\n    for d in range(3, max_divisor, 2):\n        if n % d == 0:\n            return False\n    return True\n\ndef is_palindrome(n):\n    '''Return if palindrome'''\n    return str(n) == str(n)[::-1]\n\ndef is_pandigital(n):\n    '''Return if pandigital'''\n    return set(str(n)) == set('123456789')[:len(str(n))]\n\ndef is_pandigital_0(n):\n    '''Return if pandigital'''\n    return set(str(n)) == set('0123456789')[:len(str(n))]\n\ndef is_pandigital\n```\n\n\nWhile the code of Syncode is \n```python\noutput = partial_code+syn_llm.infer(partial_code)[0]\nprint(output)\n```\nwith output \n```\nSetting `pad_token_id` to `eos_token_id`:151643 for open-end generation.\ndef is_prime(n):\n    '''Return if prime'''\n\n```\n\nIt seems the grammar constraints are masking all probable next tokens, effectively forcing the model into an early termination. Debugging with skip_special_tokens=False at [link](https://www.google.com/search?q=...\u0026authuser=1) reveals that the output degenerates into repeating special tokens (e.g., \u003c|im_start|\u003e). This may confirm that the grammar constraints are masking all valid continuation tokens, leaving the model with no valid options.","author":{"url":"https://github.com/LimFang","@type":"Person","name":"LimFang"},"datePublished":"2026-01-08T11:16:59.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":0},"url":"https://github.com/243/syncode/issues/243"}