Title: Slow when logits_all=True, inconsistent logprobs and solutions · Issue #1983 · abetlen/llama-cpp-python · GitHub
Open Graph Title: Slow when logits_all=True, inconsistent logprobs and solutions · Issue #1983 · abetlen/llama-cpp-python
X Title: Slow when logits_all=True, inconsistent logprobs and solutions · Issue #1983 · abetlen/llama-cpp-python
Description: Environment Lastest version by 2025.03.26 Issue I tried to use this library to record logits of my gguf model. But it seemed to be some errors on the return value of Llama.__call__(). And the process of recording logits is too slow. Then...
Open Graph Description: Environment Lastest version by 2025.03.26 Issue I tried to use this library to record logits of my gguf model. But it seemed to be some errors on the return value of Llama.__call__(). And the proce...
X Description: Environment Lastest version by 2025.03.26 Issue I tried to use this library to record logits of my gguf model. But it seemed to be some errors on the return value of Llama.__call__(). And the proce...
Opengraph URL: https://github.com/abetlen/llama-cpp-python/issues/1983
X: @github
Domain: github.com
{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Slow when logits_all=True, inconsistent logprobs and solutions","articleBody":"# Environment\nLastest version by 2025.03.26\n# Issue\nI tried to use this library to record logits of my gguf model. But it seemed to be some errors on the return value of Llama.\\_\\_call\\_\\_(). And the process of recording logits is too slow.\n\nThen I dived into this problem. I found that the \"token_logprobs\" in the return value of Llama.\\_\\_call\\_\\_() are not the log probabilities of the current token generated on the current position. The returned \"token_logprobs\" are actually the log probabilities of the current token generated by the model after receiving the current token.\n\nI wrote two python program to verify it.\n``` python\nfrom llama_cpp import Llama\n\nmodel_path = \"\" #gguf models\n\nprompt = \"Once upon a time,\"\n\nmodel = Llama(\n model_path= model_path,\n n_gpu_layers=-1, \n n_ctx=2048,\n logits_all=True,\n main_gpu=0,\n verbose=False\n)\n\nresponse = model(\n prompt= prompt, \n max_tokens = 1, \n echo=True,\n logprobs = 1,\n temperature = 0,\n)\n\nlogprobs = response['choices'][0]['logprobs']\ntokens = logprobs['tokens']\ntoken_logprobs = logprobs['token_logprobs']\ntop_logprobs = logprobs['top_logprobs']\n\nprint(response)\nfor i in range(len(tokens)):\n print(\"prob_ln = \", token_logprobs[i], \"Token = \", tokens[i])\n\n\nprint(len(response['choices'][0]['logprobs']['top_logprobs']))\n```\n``` python\nimport torch\nimport torch.nn.functional as F\nimport transformers\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\n\nmodel_path = \"\" # hf models.\nprompt = \"Once upon a time,\"\nmodel = AutoModelForCausalLM.from_pretrained(model_path, torch_dtype='auto', revision=\"main\")\ntokenizer = AutoTokenizer.from_pretrained(model_path, torch_dtype='auto')\n\ninputs = tokenizer(prompt, return_tensors=\"pt\")\ninput_ids = inputs[\"input_ids\"]\n\nprint(input_ids)\nwith torch.no_grad():\n outputs = model(input_ids=input_ids)\n\nlogits = F.log_softmax(outputs.logits, dim=-1)\nprint(logits)\nlogits = torch.squeeze(logits)\n\nfor i in range(logits.shape[0]): \n prob_ln = logits[i-1, input_ids[0, i]].item() #replace \"i-1\" with \"i\", you will get results same as the Llama version\n token = tokenizer.decode(input_ids[0, i]) \n print(f\"Position {i}: prob_ln = {prob_ln}, Token = '{token}', TokenID = {input_ids[0, i]}\")\n```\nIf you use the two models only different on format, you will found totaly different outputs. And you can modify the code based on my comments to verify my conclusion.\n# Solutions\nI got a rough solution for my need. But I am busy recently, unable to write thorough codes. Hope you guys can complement it to the project.\nFor the efficiency, I found a sort for a vocab size tensor in Llama.\\_create_completion( See the picture below ). But for my need, I only need one top_token_logprobs. This sort costs most time in my program. I use torch.max to fix it. For thorough need, you may add some specific optimization when the parameter \"logprobs\" is small.\nFor the inconsistent logprobs, you just need to change \"int(token)\"( See the picture below ) to the \"pre_token\".\n\nAnd here is my solution. But it only works for recording one top_tokens. You can replace them to the correspond space in llama.py if you have the same need.\n``` python\n logprobs_or_none: Optional[CompletionLogprobs] = None\n if logprobs is not None:\n text_offset = 0 if echo else len(prompt)\n token_offset = 0 if echo else len(prompt_tokens[1:])\n text_offsets: List[int] = []\n token_logprobs: List[Optional[float]] = []\n tokens: List[str] = []\n top_logprobs: List[Optional[Dict[str, float]]] = []\n\n if echo:\n # Remove leading BOS token if exists\n all_tokens = (\n prompt_tokens[1 if prompt_tokens[0] == self.token_bos() else 0 :]\n + completion_tokens\n )\n else:\n all_tokens = completion_tokens\n\n all_token_strs = [\n self.detokenize([token], prev_tokens=all_tokens[:i]).decode(\n \"utf-8\", errors=\"ignore\"\n )\n for i, token in enumerate(all_tokens)\n ]\n all_logprobs = Llama.logits_to_logprobs(self._scores)[token_offset:]\n # TODO: may be able to change this loop to use np.take_along_dim\n token_logprobs = [0.0]\n top_logprobs = [{}]\n for idx, (token, token_str, logprobs_token) in enumerate(\n zip(all_tokens, all_token_strs, all_logprobs)\n ):\n if token == bos_token_id:\n continue\n text_offsets.append(\n text_offset\n + len(\n self.detokenize(all_tokens[:idx]).decode(\n \"utf-8\", errors=\"ignore\"\n )\n )\n )\n tokens.append(token_str)\n # sorted_logprobs = list(\n # sorted(\n # zip(logprobs_token, range(len(logprobs_token))), reverse=True\n # )\n # )\n logprobs_token_tmp = torch.tensor(logprobs_token)\n max_logprob, max_indix = torch.max(logprobs_token_tmp, dim=0)\n if(idx+1 != len(all_tokens)):\n token_logprobs.append(logprobs_token[int(all_tokens[idx+1])])\n # print(token, logprobs_token[int(token)])\n top_logprob: Optional[Dict[str, float]] = {\n # self.detokenize([i], prev_tokens=all_tokens[:idx]).decode(\n # \"utf-8\", errors=\"ignore\"\n # ): logprob\n self.detokenize([max_indix], prev_tokens=all_tokens[:idx]).decode(\n \"utf-8\", errors=\"ignore\"\n ): max_logprob\n }\n # top_logprob.update({token_str: logprobs_token[int(token)]})\n top_logprobs.append(top_logprob)\n # Weird idosincracy of the OpenAI API where\n # token_logprobs and top_logprobs are null for\n # the first token.\n # if echo and len(all_tokens) > 0:\n # token_logprobs[0] = None\n # top_logprobs[0] = None\n logprobs_or_none = {\n \"tokens\": tokens,\n \"text_offset\": text_offsets,\n \"token_logprobs\": token_logprobs,\n \"top_logprobs\": top_logprobs,\n }\n```\n","author":{"url":"https://github.com/For-rest2005","@type":"Person","name":"For-rest2005"},"datePublished":"2025-03-26T04:58:06Z","url":"https://github.com/abetlen/llama-cpp-python/issues/1983"}
| react-profiling | 0 |
| react-import-map | react,react-dom,react-dom/client,react-dom/profiling,react-is,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@b8488ee2c322 |
| route-pattern | /_view_fragments/issues/show/:user_id/:repository/:id/issue_layout(.:format) |
| route-controller | voltron_issues_fragments |
| route-action | issue_layout |
| fetch-nonce | v2:e5451ce7-c4f3-5c29-19d0-53a3e58de9a8 |
| current-catalog-service-hash | 81bb79d38c15960b92d99bca9288a9108c7a47b18f2423d0f6438c5b7bcd2114 |
| request-id | C254:37308E:12E409E:1803FE7:6AFF98A4 |
| html-safe-nonce | 090e0073ea5cebec57da4206816dfa5b3bfdeba8911d1d67ea54970169c5aa42 |
| visitor-payload | eyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiJDMjU0OjM3MzA4RToxMkU0MDlFOjE4MDNGRTc6NkFGRjk4QTQiLCJ2aXNpdG9yX2lkIjoiMzk0NjU0MDU4ODQ2MzAwMzgxMiIsInJlZ2lvbl9lZGdlIjoiaWFkIiwicmVnaW9uX3JlbmRlciI6ImlhZCJ9 |
| visitor-hmac | b63d497aef4ca52f7dcf01a0744a07254fc2a3b07eaa286419f206455b21d48c |
| hovercard-subject-tag | issue:2948401219 |
| github-keyboard-shortcuts | repository,issues,copilot |
| google-site-verification | Apib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I |
| octolytics-url | https://collector.github.com/github/collect |
| analytics-location | / |
| fb:app_id | 1401488693436528 |
| apple-itunes-app | app-id=1477376905, app-argument=https://github.com/_view_fragments/issues/show/abetlen/llama-cpp-python/1983/issue_layout |
| twitter:image | https://opengraph.githubassets.com/f2b64e0620ff023e62e947807ddd393012c6f19cd932262220cd155b3f5f7a72/abetlen/llama-cpp-python/issues/1983 |
| twitter:card | summary_large_image |
| og:image | https://opengraph.githubassets.com/f2b64e0620ff023e62e947807ddd393012c6f19cd932262220cd155b3f5f7a72/abetlen/llama-cpp-python/issues/1983 |
| og:image:alt | Environment Lastest version by 2025.03.26 Issue I tried to use this library to record logits of my gguf model. But it seemed to be some errors on the return value of Llama.__call__(). And the proce... |
| og:image:width | 1200 |
| og:image:height | 600 |
| og:site_name | GitHub |
| og:type | object |
| og:author:username | For-rest2005 |
| hostname | github.com |
| expected-hostname | github.com |
| None | 2c75d045e8c84b51e2ac680086a6dd1eeaf2bc15b82137d22ee5106835d6256c |
| turbo-cache-control | no-preview |
| go-import | github.com/abetlen/llama-cpp-python git https://github.com/abetlen/llama-cpp-python.git |
| octolytics-dimension-user_id | 6826477 |
| octolytics-dimension-user_login | abetlen |
| octolytics-dimension-repository_id | 617868717 |
| octolytics-dimension-repository_nwo | abetlen/llama-cpp-python |
| octolytics-dimension-repository_public | true |
| octolytics-dimension-repository_is_fork | false |
| octolytics-dimension-repository_network_root_id | 617868717 |
| octolytics-dimension-repository_network_root_nwo | abetlen/llama-cpp-python |
| turbo-body-classes | logged-out env-production page-responsive |
| disable-turbo | false |
| browser-stats-url | https://api.github.com/_private/browser/stats |
| browser-errors-url | https://api.github.com/_private/browser/errors |
| release | 525b66d22c4f7e6bb8b62934bec198964f7e8b93 |
| ui-target | full |
| theme-color | #1e2327 |
| color-scheme | light dark |
| react-app-name | issues |
Links:
Viewport: width=device-width