René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Prevent starchat from generating nonsense after ouptutting \u003c|system|\u003e \u003c|end|\u003e \u003c|user|\u003e","articleBody":"When using starchat the model will likely start talking bullshit (in Spanish) after printing out the sequence:\r\n\r\n```\r\n\u003c|system|\u003e \u003c|end|\u003e \u003c|user|\u003e\r\n```\r\n\r\nI added a rudimentary fix to stop generating new tokens in case starchat is used after outputting \u003c|system|\u003e \u003c|end|\u003e \u003c|user|\u003e:\r\n\r\n```\r\n        // check if model is santacoder\r\n        if (model.hparams.n_layer \u003c= 30 \u0026\u0026 embd.back() == 49152) {\r\n            break;\r\n        }\r\n        // check if model is starcoder\r\n        else if (embd.back() == 0) { //TODO: this is only for starcoder\r\n            break;\r\n        }\r\n        // starchat since only these 3 models are supported atm\r\n        else{\r\n            // break to prevent starchat from talking gibberish\r\n            if (output.find(\"\u003c|system|\u003e\\n\u003c|end|\u003e\\n\u003c|user|\u003e\") != std::string::npos) {\r\n                break;\r\n            }\r\n        }\r\n```\r\n\r\nWould that be desirable pr? And also what is the correct way to detect if the model is indeed starcoder instead of using the default?","author":{"url":"https://github.com/KukumavMozolo","@type":"Person","name":"KukumavMozolo"},"datePublished":"2023-08-22T08:10:43.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":0},"url":"https://github.com/32/starcoder.cpp/issues/32"}