Title: [BUG] Server cant handle two streaming connections in same time · Issue #897 · abetlen/llama-cpp-python · GitHub
Open Graph Title: [BUG] Server cant handle two streaming connections in same time · Issue #897 · abetlen/llama-cpp-python
X Title: [BUG] Server cant handle two streaming connections in same time · Issue #897 · abetlen/llama-cpp-python
Description: Prerequisites Please answer the following questions for yourself before submitting an issue. I am running the latest code. Development is very rapid so there are no tagged versions as of now. I carefully followed the README.md. I searche...
Open Graph Description: Prerequisites Please answer the following questions for yourself before submitting an issue. I am running the latest code. Development is very rapid so there are no tagged versions as of now. I car...
X Description: Prerequisites Please answer the following questions for yourself before submitting an issue. I am running the latest code. Development is very rapid so there are no tagged versions as of now. I car...
Opengraph URL: https://github.com/abetlen/llama-cpp-python/issues/897
X: @github
Domain: github.com
{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"[BUG] Server cant handle two streaming connections in same time","articleBody":"# Prerequisites\r\n\r\nPlease answer the following questions for yourself before submitting an issue.\r\n\r\n- [x] I am running the latest code. Development is very rapid so there are no tagged versions as of now.\r\n- [x] I carefully followed the [README.md](https://github.com/abetlen/llama-cpp-python/blob/main/README.md).\r\n- [x] I [searched using keywords relevant to my issue](https://docs.github.com/en/issues/tracking-your-work-with-issues/filtering-and-searching-issues-and-pull-requests) to make sure that I am creating a new issue that is not already open (or closed).\r\n- [x] I reviewed the [Discussions](https://github.com/abetlen/llama-cpp-python/discussions), and have a new bug or useful enhancement to share.\r\n\r\n# Expected Behavior\r\n\r\n1. Start llama-cpp-server in docker container\r\n2. Open 2 terminals\r\n3. In each start streaming completion sessinon\r\n4. Either each chunk of both streams processed and returned (each terminal get streaming response in same time)\r\n5. Or firstly one terminal get all full result, and after another terminal get all full result (as its working now for not-streaming)\r\n\r\n# Current Behavior\r\n\r\n1. Start llama-cpp-server in docker container\r\n2. Open 2 terminals\r\n3. In each start streaming completion sessinon\r\n4. First terminal crashed with error `ValueError: invalid literal for int() with base 16: b''`\r\n5. Second terminal not crashed and get streaming response.\r\n\r\n# Environment and Context\r\n\r\n* Physical (or virtual) hardware you are using, e.g. for Linux:\r\n\r\n```\r\nArchitecture: x86_64\r\n CPU op-mode(s): 32-bit, 64-bit\r\n Address sizes: 39 bits physical, 48 bits virtual\r\n Byte Order: Little Endian\r\nCPU(s): 20\r\n On-line CPU(s) list: 0-19\r\nVendor ID: GenuineIntel\r\n Model name: 12th Gen Intel(R) Core(TM) i7-12700F\r\n CPU family: 6\r\n Model: 151\r\n Thread(s) per core: 2\r\n Core(s) per socket: 12\r\n Socket(s): 1\r\n Stepping: 2\r\n CPU max MHz: 4900.0000\r\n CPU min MHz: 800.0000\r\n BogoMIPS: 4224.00\r\n Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmu\r\n lqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 sdbg fma cx16 xtpr pdcm sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault ssbd ibrs ibpb stibp ibrs_enhanced tpr_shadow vnmi flexpriority ept vpid ept_ad\r\n fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid rdseed adx smap clflushopt clwb intel_pt sha_ni xsaveopt xsavec xgetbv1 xsaves split_lock_detect avx_vnni dtherm ida arat pln pts hwp hwp_notify hwp_act_window hwp_epp hwp_pkg_req umip pku ospke waitpkg gfni vae\r\n s vpclmulqdq rdpid movdiri movdir64b fsrm md_clear serialize arch_lbr flush_l1d arch_capabilities\r\nVirtualization features: \r\n Virtualization: VT-x\r\nCaches (sum of all): \r\n L1d: 512 KiB (12 instances)\r\n L1i: 512 KiB (12 instances)\r\n L2: 12 MiB (9 instances)\r\n L3: 25 MiB (1 instance)\r\nNUMA: \r\n NUMA node(s): 1\r\n NUMA node0 CPU(s): 0-19\r\nVulnerabilities: \r\n Gather data sampling: Not affected\r\n Itlb multihit: Not affected\r\n L1tf: Not affected\r\n Mds: Not affected\r\n Meltdown: Not affected\r\n Mmio stale data: Not affected\r\n Retbleed: Not affected\r\n Spec rstack overflow: Not affected\r\n Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp\r\n Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization\r\n Spectre v2: Mitigation; Enhanced IBRS, IBPB conditional, RSB filling, PBRSB-eIBRS SW sequence\r\n Srbds: Not affected\r\n Tsx async abort: Not affected\r\n```\r\n\r\n* Operating System, e.g. for Linux:\r\n\r\n```\r\nLinux gpu-serv-2 5.15.0-86-generic #96-Ubuntu SMP Wed Sep 20 08:23:49 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux\r\n```\r\n\r\n\r\n* SDK version:\r\n\r\n```\r\nmain branch \r\nhttps://github.com/abetlen/llama-cpp-python/commit/82072802ea0eb68f7f226425e5ea434a3e8e60a0\r\n```\r\n\r\n# Failure Information (for bugs)\r\n\r\nError on client side (first terminal):\r\n```\r\ntext = \"12345 67890 10Traceback (most recent call last):\r\n File \"/home/kuruhuru/micromamba/envs/promptflow/lib/python3.9/site-packages/urllib3/response.py\", line 761, in _update_chunk_length\r\n self.chunk_left = int(line, 16)\r\nValueError: invalid literal for int() with base 16: b''\r\n\r\nDuring handling of the above exception, another exception occurred:\r\n\r\nTraceback (most recent call last):\r\n File \"/home/kuruhuru/micromamba/envs/promptflow/lib/python3.9/site-packages/urllib3/response.py\", line 444, in _error_catcher\r\n yield\r\n File \"/home/kuruhuru/micromamba/envs/promptflow/lib/python3.9/site-packages/urllib3/response.py\", line 828, in read_chunked\r\n self._update_chunk_length()\r\n File \"/home/kuruhuru/micromamba/envs/promptflow/lib/python3.9/site-packages/urllib3/response.py\", line 765, in _update_chunk_length\r\n raise InvalidChunkLength(self, line)\r\nurllib3.exceptions.InvalidChunkLength: InvalidChunkLength(got length b'', 0 bytes read)\r\n\r\nDuring handling of the above exception, another exception occurred:\r\n\r\nTraceback (most recent call last):\r\n File \"/home/kuruhuru/micromamba/envs/promptflow/lib/python3.9/site-packages/requests/models.py\", line 816, in generate\r\n yield from self.raw.stream(chunk_size, decode_content=True)\r\n File \"/home/kuruhuru/micromamba/envs/promptflow/lib/python3.9/site-packages/urllib3/response.py\", line 624, in stream\r\n for line in self.read_chunked(amt, decode_content=decode_content):\r\n File \"/home/kuruhuru/micromamba/envs/promptflow/lib/python3.9/site-packages/urllib3/response.py\", line 857, in read_chunked\r\n self._original_response.close()\r\n File \"/home/kuruhuru/micromamba/envs/promptflow/lib/python3.9/contextlib.py\", line 137, in __exit__\r\n self.gen.throw(typ, value, traceback)\r\n File \"/home/kuruhuru/micromamba/envs/promptflow/lib/python3.9/site-packages/urllib3/response.py\", line 461, in _error_catcher\r\n raise ProtocolError(\"Connection broken: %r\" % e, e)\r\nurllib3.exceptions.ProtocolError: (\"Connection broken: InvalidChunkLength(got length b'', 0 bytes read)\", InvalidChunkLength(got length b'', 0 bytes read))\r\n\r\nDuring handling of the above exception, another exception occurred:\r\n\r\nTraceback (most recent call last):\r\n File \"/home/kuruhuru/dev/llm/gguf-server/examples/streaming.py\", line 21, in \u003cmodule>\r\n for a in response:\r\n File \"/home/kuruhuru/micromamba/envs/promptflow/lib/python3.9/site-packages/openai/api_resources/abstract/engine_api_resource.py\", line 166, in \u003cgenexpr>\r\n return (\r\n File \"/home/kuruhuru/micromamba/envs/promptflow/lib/python3.9/site-packages/openai/api_requestor.py\", line 692, in \u003cgenexpr>\r\n return (\r\n File \"/home/kuruhuru/micromamba/envs/promptflow/lib/python3.9/site-packages/openai/api_requestor.py\", line 115, in parse_stream\r\n for line in rbody:\r\n File \"/home/kuruhuru/micromamba/envs/promptflow/lib/python3.9/site-packages/requests/models.py\", line 865, in iter_lines\r\n for chunk in self.iter_content(\r\n File \"/home/kuruhuru/micromamba/envs/promptflow/lib/python3.9/site-packages/requests/models.py\", line 818, in generate\r\n raise ChunkedEncodingError(e)\r\nrequests.exceptions.ChunkedEncodingError: (\"Connection broken: InvalidChunkLength(got length b'', 0 bytes read)\", InvalidChunkLength(got length b'', 0 bytes read))\r\n```\r\n\r\nError on server side:\r\n```\r\nERROR: ASGI callable returned without completing response.\r\nLlama.generate: prefix-match hit\r\ndisconnected\r\nDisconnected from client (via refresh/close) Address(host='172.19.0.1', port=53080)\r\n```","author":{"url":"https://github.com/ArtyomZemlyak","@type":"Person","name":"ArtyomZemlyak"},"datePublished":"2023-11-10T02:58:37Z","url":"https://github.com/abetlen/llama-cpp-python/issues/897"}
| react-profiling | 0 |
| react-import-map | react,react-dom,react-dom/client,react-dom/profiling,react-is,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@b8488ee2c322 |
| route-pattern | /_view_fragments/issues/show/:user_id/:repository/:id/issue_layout(.:format) |
| route-controller | voltron_issues_fragments |
| route-action | issue_layout |
| fetch-nonce | v2:fefcaa7c-3baf-febd-1334-062d7eac995e |
| current-catalog-service-hash | 81bb79d38c15960b92d99bca9288a9108c7a47b18f2423d0f6438c5b7bcd2114 |
| request-id | 9048:3677D1:5A6E43:74A3C1:6AFFC439 |
| html-safe-nonce | d863ceeb4351a8c4698c2287e54872403ef51df7ca04775a35f5c84a7a592af4 |
| visitor-payload | eyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiI5MDQ4OjM2NzdEMTo1QTZFNDM6NzRBM0MxOjZBRkZDNDM5IiwidmlzaXRvcl9pZCI6IjI2MTM4MjU0MzkyNDM2ODI4NzMiLCJyZWdpb25fZWRnZSI6ImlhZCIsInJlZ2lvbl9yZW5kZXIiOiJpYWQifQ== |
| visitor-hmac | 92260ece5173482d15206ea0c9ff8db273e4da75f21d3ce29b460bd4a90a55e8 |
| hovercard-subject-tag | issue:1986758823 |
| github-keyboard-shortcuts | repository,issues,copilot |
| google-site-verification | Apib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I |
| octolytics-url | https://collector.github.com/github/collect |
| analytics-location | / |
| fb:app_id | 1401488693436528 |
| apple-itunes-app | app-id=1477376905, app-argument=https://github.com/_view_fragments/issues/show/abetlen/llama-cpp-python/897/issue_layout |
| twitter:image | https://opengraph.githubassets.com/72c52038d282fe0cfcbf152d87bae840b34018f680459bbeaea99ec69d32fb24/abetlen/llama-cpp-python/issues/897 |
| twitter:card | summary_large_image |
| og:image | https://opengraph.githubassets.com/72c52038d282fe0cfcbf152d87bae840b34018f680459bbeaea99ec69d32fb24/abetlen/llama-cpp-python/issues/897 |
| og:image:alt | Prerequisites Please answer the following questions for yourself before submitting an issue. I am running the latest code. Development is very rapid so there are no tagged versions as of now. I car... |
| og:image:width | 1200 |
| og:image:height | 600 |
| og:site_name | GitHub |
| og:type | object |
| og:author:username | ArtyomZemlyak |
| hostname | github.com |
| expected-hostname | github.com |
| None | 1e805b832df235bb21305db712c3b6432920019dfc12cb25b9cb8d3b0c9403cf |
| turbo-cache-control | no-preview |
| go-import | github.com/abetlen/llama-cpp-python git https://github.com/abetlen/llama-cpp-python.git |
| octolytics-dimension-user_id | 6826477 |
| octolytics-dimension-user_login | abetlen |
| octolytics-dimension-repository_id | 617868717 |
| octolytics-dimension-repository_nwo | abetlen/llama-cpp-python |
| octolytics-dimension-repository_public | true |
| octolytics-dimension-repository_is_fork | false |
| octolytics-dimension-repository_network_root_id | 617868717 |
| octolytics-dimension-repository_network_root_nwo | abetlen/llama-cpp-python |
| turbo-body-classes | logged-out env-production page-responsive |
| disable-turbo | false |
| browser-stats-url | https://api.github.com/_private/browser/stats |
| browser-errors-url | https://api.github.com/_private/browser/errors |
| release | 8fa4a54e6871c7908eda1c246ad5972982c0b59d |
| ui-target | full |
| theme-color | #1e2327 |
| color-scheme | light dark |
| react-app-name | issues |
Links:
Viewport: width=device-width