René's URL Explorer Experiment


Title: [BUG] Server cant handle two streaming connections in same time · Issue #897 · abetlen/llama-cpp-python · GitHub

Open Graph Title: [BUG] Server cant handle two streaming connections in same time · Issue #897 · abetlen/llama-cpp-python

X Title: [BUG] Server cant handle two streaming connections in same time · Issue #897 · abetlen/llama-cpp-python

Description: Prerequisites Please answer the following questions for yourself before submitting an issue. I am running the latest code. Development is very rapid so there are no tagged versions as of now. I carefully followed the README.md. I searche...

Open Graph Description: Prerequisites Please answer the following questions for yourself before submitting an issue. I am running the latest code. Development is very rapid so there are no tagged versions as of now. I car...

X Description: Prerequisites Please answer the following questions for yourself before submitting an issue. I am running the latest code. Development is very rapid so there are no tagged versions as of now. I car...

Opengraph URL: https://github.com/abetlen/llama-cpp-python/issues/897

X: @github

direct link

Domain: github.com


Hey, it has json ld scripts:
{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"[BUG] Server cant handle two streaming connections in same time","articleBody":"# Prerequisites\r\n\r\nPlease answer the following questions for yourself before submitting an issue.\r\n\r\n- [x] I am running the latest code. Development is very rapid so there are no tagged versions as of now.\r\n- [x] I carefully followed the [README.md](https://github.com/abetlen/llama-cpp-python/blob/main/README.md).\r\n- [x] I [searched using keywords relevant to my issue](https://docs.github.com/en/issues/tracking-your-work-with-issues/filtering-and-searching-issues-and-pull-requests) to make sure that I am creating a new issue that is not already open (or closed).\r\n- [x] I reviewed the [Discussions](https://github.com/abetlen/llama-cpp-python/discussions), and have a new bug or useful enhancement to share.\r\n\r\n# Expected Behavior\r\n\r\n1. Start llama-cpp-server in docker container\r\n2. Open 2 terminals\r\n3. In each start streaming completion sessinon\r\n4. Either each chunk of both streams processed and returned (each terminal get streaming response in same time)\r\n5. Or firstly one terminal get all full result, and after another terminal get all full result (as its working now for not-streaming)\r\n\r\n# Current Behavior\r\n\r\n1. Start llama-cpp-server in docker container\r\n2. Open 2 terminals\r\n3. In each start streaming completion sessinon\r\n4. First terminal crashed with error `ValueError: invalid literal for int() with base 16: b''`\r\n5. Second terminal not crashed and get streaming response.\r\n\r\n# Environment and Context\r\n\r\n* Physical (or virtual) hardware you are using, e.g. for Linux:\r\n\r\n```\r\nArchitecture:            x86_64\r\n  CPU op-mode(s):        32-bit, 64-bit\r\n  Address sizes:         39 bits physical, 48 bits virtual\r\n  Byte Order:            Little Endian\r\nCPU(s):                  20\r\n  On-line CPU(s) list:   0-19\r\nVendor ID:               GenuineIntel\r\n  Model name:            12th Gen Intel(R) Core(TM) i7-12700F\r\n    CPU family:          6\r\n    Model:               151\r\n    Thread(s) per core:  2\r\n    Core(s) per socket:  12\r\n    Socket(s):           1\r\n    Stepping:            2\r\n    CPU max MHz:         4900.0000\r\n    CPU min MHz:         800.0000\r\n    BogoMIPS:            4224.00\r\n    Flags:               fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmu\r\n                         lqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 sdbg fma cx16 xtpr pdcm sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault ssbd ibrs ibpb stibp ibrs_enhanced tpr_shadow vnmi flexpriority ept vpid ept_ad\r\n                          fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid rdseed adx smap clflushopt clwb intel_pt sha_ni xsaveopt xsavec xgetbv1 xsaves split_lock_detect avx_vnni dtherm ida arat pln pts hwp hwp_notify hwp_act_window hwp_epp hwp_pkg_req umip pku ospke waitpkg gfni vae\r\n                         s vpclmulqdq rdpid movdiri movdir64b fsrm md_clear serialize arch_lbr flush_l1d arch_capabilities\r\nVirtualization features: \r\n  Virtualization:        VT-x\r\nCaches (sum of all):     \r\n  L1d:                   512 KiB (12 instances)\r\n  L1i:                   512 KiB (12 instances)\r\n  L2:                    12 MiB (9 instances)\r\n  L3:                    25 MiB (1 instance)\r\nNUMA:                    \r\n  NUMA node(s):          1\r\n  NUMA node0 CPU(s):     0-19\r\nVulnerabilities:         \r\n  Gather data sampling:  Not affected\r\n  Itlb multihit:         Not affected\r\n  L1tf:                  Not affected\r\n  Mds:                   Not affected\r\n  Meltdown:              Not affected\r\n  Mmio stale data:       Not affected\r\n  Retbleed:              Not affected\r\n  Spec rstack overflow:  Not affected\r\n  Spec store bypass:     Mitigation; Speculative Store Bypass disabled via prctl and seccomp\r\n  Spectre v1:            Mitigation; usercopy/swapgs barriers and __user pointer sanitization\r\n  Spectre v2:            Mitigation; Enhanced IBRS, IBPB conditional, RSB filling, PBRSB-eIBRS SW sequence\r\n  Srbds:                 Not affected\r\n  Tsx async abort:       Not affected\r\n```\r\n\r\n* Operating System, e.g. for Linux:\r\n\r\n```\r\nLinux gpu-serv-2 5.15.0-86-generic #96-Ubuntu SMP Wed Sep 20 08:23:49 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux\r\n```\r\n\r\n\r\n* SDK version:\r\n\r\n```\r\nmain branch \r\nhttps://github.com/abetlen/llama-cpp-python/commit/82072802ea0eb68f7f226425e5ea434a3e8e60a0\r\n```\r\n\r\n# Failure Information (for bugs)\r\n\r\nError on client side (first terminal):\r\n```\r\ntext = \"12345 67890 10Traceback (most recent call last):\r\n  File \"/home/kuruhuru/micromamba/envs/promptflow/lib/python3.9/site-packages/urllib3/response.py\", line 761, in _update_chunk_length\r\n    self.chunk_left = int(line, 16)\r\nValueError: invalid literal for int() with base 16: b''\r\n\r\nDuring handling of the above exception, another exception occurred:\r\n\r\nTraceback (most recent call last):\r\n  File \"/home/kuruhuru/micromamba/envs/promptflow/lib/python3.9/site-packages/urllib3/response.py\", line 444, in _error_catcher\r\n    yield\r\n  File \"/home/kuruhuru/micromamba/envs/promptflow/lib/python3.9/site-packages/urllib3/response.py\", line 828, in read_chunked\r\n    self._update_chunk_length()\r\n  File \"/home/kuruhuru/micromamba/envs/promptflow/lib/python3.9/site-packages/urllib3/response.py\", line 765, in _update_chunk_length\r\n    raise InvalidChunkLength(self, line)\r\nurllib3.exceptions.InvalidChunkLength: InvalidChunkLength(got length b'', 0 bytes read)\r\n\r\nDuring handling of the above exception, another exception occurred:\r\n\r\nTraceback (most recent call last):\r\n  File \"/home/kuruhuru/micromamba/envs/promptflow/lib/python3.9/site-packages/requests/models.py\", line 816, in generate\r\n    yield from self.raw.stream(chunk_size, decode_content=True)\r\n  File \"/home/kuruhuru/micromamba/envs/promptflow/lib/python3.9/site-packages/urllib3/response.py\", line 624, in stream\r\n    for line in self.read_chunked(amt, decode_content=decode_content):\r\n  File \"/home/kuruhuru/micromamba/envs/promptflow/lib/python3.9/site-packages/urllib3/response.py\", line 857, in read_chunked\r\n    self._original_response.close()\r\n  File \"/home/kuruhuru/micromamba/envs/promptflow/lib/python3.9/contextlib.py\", line 137, in __exit__\r\n    self.gen.throw(typ, value, traceback)\r\n  File \"/home/kuruhuru/micromamba/envs/promptflow/lib/python3.9/site-packages/urllib3/response.py\", line 461, in _error_catcher\r\n    raise ProtocolError(\"Connection broken: %r\" % e, e)\r\nurllib3.exceptions.ProtocolError: (\"Connection broken: InvalidChunkLength(got length b'', 0 bytes read)\", InvalidChunkLength(got length b'', 0 bytes read))\r\n\r\nDuring handling of the above exception, another exception occurred:\r\n\r\nTraceback (most recent call last):\r\n  File \"/home/kuruhuru/dev/llm/gguf-server/examples/streaming.py\", line 21, in \u003cmodule>\r\n    for a in response:\r\n  File \"/home/kuruhuru/micromamba/envs/promptflow/lib/python3.9/site-packages/openai/api_resources/abstract/engine_api_resource.py\", line 166, in \u003cgenexpr>\r\n    return (\r\n  File \"/home/kuruhuru/micromamba/envs/promptflow/lib/python3.9/site-packages/openai/api_requestor.py\", line 692, in \u003cgenexpr>\r\n    return (\r\n  File \"/home/kuruhuru/micromamba/envs/promptflow/lib/python3.9/site-packages/openai/api_requestor.py\", line 115, in parse_stream\r\n    for line in rbody:\r\n  File \"/home/kuruhuru/micromamba/envs/promptflow/lib/python3.9/site-packages/requests/models.py\", line 865, in iter_lines\r\n    for chunk in self.iter_content(\r\n  File \"/home/kuruhuru/micromamba/envs/promptflow/lib/python3.9/site-packages/requests/models.py\", line 818, in generate\r\n    raise ChunkedEncodingError(e)\r\nrequests.exceptions.ChunkedEncodingError: (\"Connection broken: InvalidChunkLength(got length b'', 0 bytes read)\", InvalidChunkLength(got length b'', 0 bytes read))\r\n```\r\n\r\nError on server side:\r\n```\r\nERROR:    ASGI callable returned without completing response.\r\nLlama.generate: prefix-match hit\r\ndisconnected\r\nDisconnected from client (via refresh/close) Address(host='172.19.0.1', port=53080)\r\n```","author":{"url":"https://github.com/ArtyomZemlyak","@type":"Person","name":"ArtyomZemlyak"},"datePublished":"2023-11-10T02:58:37Z","url":"https://github.com/abetlen/llama-cpp-python/issues/897"}

react-profiling0
react-import-mapreact,react-dom,react-dom/client,react-dom/profiling,react-is,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@b8488ee2c322
route-pattern/_view_fragments/issues/show/:user_id/:repository/:id/issue_layout(.:format)
route-controllervoltron_issues_fragments
route-actionissue_layout
fetch-noncev2:fefcaa7c-3baf-febd-1334-062d7eac995e
current-catalog-service-hash81bb79d38c15960b92d99bca9288a9108c7a47b18f2423d0f6438c5b7bcd2114
request-id9048:3677D1:5A6E43:74A3C1:6AFFC439
html-safe-nonced863ceeb4351a8c4698c2287e54872403ef51df7ca04775a35f5c84a7a592af4
visitor-payloadeyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiI5MDQ4OjM2NzdEMTo1QTZFNDM6NzRBM0MxOjZBRkZDNDM5IiwidmlzaXRvcl9pZCI6IjI2MTM4MjU0MzkyNDM2ODI4NzMiLCJyZWdpb25fZWRnZSI6ImlhZCIsInJlZ2lvbl9yZW5kZXIiOiJpYWQifQ==
visitor-hmac92260ece5173482d15206ea0c9ff8db273e4da75f21d3ce29b460bd4a90a55e8
hovercard-subject-tagissue:1986758823
github-keyboard-shortcutsrepository,issues,copilot
google-site-verificationApib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I
octolytics-urlhttps://collector.github.com/github/collect
analytics-location///voltron/issues_fragments/issue_layout
fb:app_id1401488693436528
apple-itunes-appapp-id=1477376905, app-argument=https://github.com/_view_fragments/issues/show/abetlen/llama-cpp-python/897/issue_layout
twitter:imagehttps://opengraph.githubassets.com/72c52038d282fe0cfcbf152d87bae840b34018f680459bbeaea99ec69d32fb24/abetlen/llama-cpp-python/issues/897
twitter:cardsummary_large_image
og:imagehttps://opengraph.githubassets.com/72c52038d282fe0cfcbf152d87bae840b34018f680459bbeaea99ec69d32fb24/abetlen/llama-cpp-python/issues/897
og:image:altPrerequisites Please answer the following questions for yourself before submitting an issue. I am running the latest code. Development is very rapid so there are no tagged versions as of now. I car...
og:image:width1200
og:image:height600
og:site_nameGitHub
og:typeobject
og:author:usernameArtyomZemlyak
hostnamegithub.com
expected-hostnamegithub.com
None1e805b832df235bb21305db712c3b6432920019dfc12cb25b9cb8d3b0c9403cf
turbo-cache-controlno-preview
go-importgithub.com/abetlen/llama-cpp-python git https://github.com/abetlen/llama-cpp-python.git
octolytics-dimension-user_id6826477
octolytics-dimension-user_loginabetlen
octolytics-dimension-repository_id617868717
octolytics-dimension-repository_nwoabetlen/llama-cpp-python
octolytics-dimension-repository_publictrue
octolytics-dimension-repository_is_forkfalse
octolytics-dimension-repository_network_root_id617868717
octolytics-dimension-repository_network_root_nwoabetlen/llama-cpp-python
turbo-body-classeslogged-out env-production page-responsive
disable-turbofalse
browser-stats-urlhttps://api.github.com/_private/browser/stats
browser-errors-urlhttps://api.github.com/_private/browser/errors
release8fa4a54e6871c7908eda1c246ad5972982c0b59d
ui-targetfull
theme-color#1e2327
color-schemelight dark
react-app-nameissues

Links:

Skip to contenthttps://github.com/abetlen/llama-cpp-python/issues/897#start-of-content
https://github.com/
Sign inhttps://github.com/login?return_to=https%3A%2F%2Fgithub.com%2Fabetlen%2Fllama-cpp-python%2Fissues%2F897
GitHubRepository hosting, code review, and collaborationhttps://github.com/platform
CopilotAI coding agent for tasks, projects, and automated workhttps://github.com/features/copilot
ActionsCI/CD and automationhttps://github.com/features/actions
Code QualityQuality analysis, fixes, and enforcementhttps://github.com/features/code-quality
CodespacesCloud development environmentshttps://github.com/features/codespaces
Code SecurityVulnerability detection and remediationhttps://github.com/security/advanced-security/code-security
Secret ProtectionSecret scanning and leak preventionhttps://github.com/security/advanced-security/secret-protection
All Featureshttps://github.com/features
Documentationhttps://docs.github.com/en
Traininghttps://learn.github.com/skills
Certificationshttps://learn.github.com/credentials
Build with Copilothttps://github.com/features/ai
Customize Copilothttps://docs.github.com/en/copilot/how-tos/copilot-on-github/customize-copilot/customize-copilot-overview
Integrationshttps://github.com/marketplace
Trending reposhttps://github.com/trending
GitHub Sponsorshttps://github.com/open-source/sponsors
Developer articleshttps://github.com/resources/articles
ChangelogYour source for the latest features, improvements, and fixes across GitHub products, all in one place.https://github.blog/changelog/
Enterpriseshttps://github.com/enterprise
Teamhttps://github.com/team
Startupshttps://github.com/enterprise/startups
Nonprofitshttps://github.com/solutions/industry/nonprofits
Healthcarehttps://github.com/solutions/industry/healthcare
Financial serviceshttps://github.com/solutions/industry/financial-services
Manufacturinghttps://github.com/solutions/industry/manufacturing
Governmenthttps://github.com/solutions/industry/government
See all industrieshttps://github.com/solutions/industry
Why GitHubhttps://github.com/why-github
Customer Storieshttps://github.com/customer-stories
Business insightshttps://github.com/solutions/executive-insights
Trust Centerhttps://github.com/trust-center
GitHub Advanced SecurityEnterprise-grade code securityhttps://github.com/security/advanced-security
Copilot for businessEnterprise-grade AI featureshttps://github.com/features/copilot/copilot-business
Premium SupportEnterprise-grade 24/7 supporthttps://github.com/enterprise/premium-support
Documentationhttps://docs.github.com/en
Resource centerhttps://github.com/resources
Bloghttps://github.blog/
Events & webinarshttps://github.com/resources/events
Community Forumhttps://github.com/orgs/community/discussions
Customer Storieshttps://github.com/customer-stories
Integrationshttps://github.com/marketplace
Customer supporthttps://help.github.com/
Traininghttps://learn.github.com/skills
GitHub Statushttps://www.githubstatus.com/
Abouthttps://github.com/about
Partnershttps://github.com/partners
Changeloghttps://github.blog/changelog/
GitHub UniverseThe event for developers, by developers — talks, workshops, and what’s next for GitHub.https://githubuniverse.com/
Pricinghttps://github.com/pricing
Sign inhttps://github.com/login?return_to=https%3A%2F%2Fgithub.com%2Fabetlen%2Fllama-cpp-python%2Fissues%2F897
Sign uphttps://github.com/signup?ref_cta=Sign+up&ref_loc=header+logged+out&ref_page=%2F%3Cuser-name%3E%2F%3Crepo-name%3E%2Fvoltron%2Fissues_fragments%2Fissue_layout&source=header-repo&source_repo=abetlen%2Fllama-cpp-python
Reloadhttps://github.com/abetlen/llama-cpp-python/issues/897
Reloadhttps://github.com/abetlen/llama-cpp-python/issues/897
Reloadhttps://github.com/abetlen/llama-cpp-python/issues/897
abetlenhttps://github.com/abetlen
llama-cpp-pythonhttps://github.com/abetlen/llama-cpp-python
Codehttps://github.com/abetlen/llama-cpp-python
Issues600 (600)https://github.com/abetlen/llama-cpp-python/issues
Pull requests122 (122)https://github.com/abetlen/llama-cpp-python/pulls
Discussionshttps://github.com/abetlen/llama-cpp-python/discussions
Actionshttps://github.com/abetlen/llama-cpp-python/actions
Projectshttps://github.com/abetlen/llama-cpp-python/projects
Security and quality1 (1)https://github.com/abetlen/llama-cpp-python/security
Insightshttps://github.com/abetlen/llama-cpp-python/pulse
https://github.com/abetlen/llama-cpp-python
New issuehttps://github.com/login?return_to=https://github.com/abetlen/llama-cpp-python/issues/897
New issuehttps://github.com/login?return_to=https://github.com/abetlen/llama-cpp-python/issues/897
[BUG] Server cant handle two streaming connections in same timehttps://github.com/abetlen/llama-cpp-python/issues/897#top
bugSomething isn't workinghttps://github.com/abetlen/llama-cpp-python/issues?q=state%3Aopen%20label%3A%22bug%22
documentationImprovements or additions to documentationhttps://github.com/abetlen/llama-cpp-python/issues?q=state%3Aopen%20label%3A%22documentation%22
https://github.com/ArtyomZemlyak
ArtyomZemlyakhttps://github.com/ArtyomZemlyak
on Nov 10, 2023https://github.com/abetlen/llama-cpp-python/issues/897#issue-1986758823
README.mdhttps://github.com/abetlen/llama-cpp-python/blob/main/README.md
searched using keywords relevant to my issuehttps://docs.github.com/en/issues/tracking-your-work-with-issues/filtering-and-searching-issues-and-pull-requests
Discussionshttps://github.com/abetlen/llama-cpp-python/discussions
https://github.com/ArtyomZemlyak
ArtyomZemlyakhttps://github.com/ArtyomZemlyak
on Nov 10, 2023https://github.com/abetlen/llama-cpp-python/issues/897#issuecomment-1805001715
microsoft/promptflow#1052https://github.com/microsoft/promptflow/issues/1052
https://github.com/ArtyomZemlyak
ArtyomZemlyakhttps://github.com/ArtyomZemlyak
on Nov 10, 2023https://github.com/abetlen/llama-cpp-python/issues/897#issuecomment-1805025721
llama-cpp-python/llama_cpp/server/app.pyhttps://github.com/abetlen/llama-cpp-python/blob/82072802ea0eb68f7f226425e5ea434a3e8e60a0/llama_cpp/server/app.py#L165-L168
8207280https://github.com/abetlen/llama-cpp-python/commit/82072802ea0eb68f7f226425e5ea434a3e8e60a0
https://github.com/abetlen
abetlenhttps://github.com/abetlen
on Nov 10, 2023https://github.com/abetlen/llama-cpp-python/issues/897#issuecomment-1806022045
@ArtyomZemlyakhttps://github.com/ArtyomZemlyak
#771https://github.com/abetlen/llama-cpp-python/issues/771
abetlenhttps://github.com/abetlen
bugSomething isn't workinghttps://github.com/abetlen/llama-cpp-python/issues?q=state%3Aopen%20label%3A%22bug%22
documentationImprovements or additions to documentationhttps://github.com/abetlen/llama-cpp-python/issues?q=state%3Aopen%20label%3A%22documentation%22
on Dec 22, 2023https://github.com/abetlen/llama-cpp-python/issues/897#event-11329419107
https://github.com/wac81
wac81https://github.com/wac81
on Feb 14, 2024https://github.com/abetlen/llama-cpp-python/issues/897#issuecomment-1944391040
@ArtyomZemlyakhttps://github.com/ArtyomZemlyak
#771https://github.com/abetlen/llama-cpp-python/issues/771
https://github.com/celsowm
celsowmhttps://github.com/celsowm
on Feb 11, 2025https://github.com/abetlen/llama-cpp-python/issues/897#issuecomment-2651706015
@abetlenhttps://github.com/abetlen
Sign up for freehttps://github.com/signup?return_to=https://github.com/abetlen/llama-cpp-python/issues/897
Sign in to commenthttps://github.com/login?return_to=https://github.com/abetlen/llama-cpp-python/issues/897
bugSomething isn't workinghttps://github.com/abetlen/llama-cpp-python/issues?q=state%3Aopen%20label%3A%22bug%22
documentationImprovements or additions to documentationhttps://github.com/abetlen/llama-cpp-python/issues?q=state%3Aopen%20label%3A%22documentation%22
https://github.com
Termshttps://docs.github.com/site-policy/github-terms/github-terms-of-service
Privacyhttps://docs.github.com/site-policy/privacy-policies/github-privacy-statement
Securityhttps://github.com/security
Statushttps://www.githubstatus.com/
Communityhttps://github.community/
Docshttps://docs.github.com/
Contacthttps://support.github.com?tags=dotcom-footer

Viewport: width=device-width


URLs of crawlers that visited me.