René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Setting the Flow Control MaxOutstandingElementCount cause stuck messages","articleBody":"The behaviour was observed in the 0.21.1-beta version of the Java client library\r\nIt seems resolved in the 0.23.1-beta version, however this newer version implements the bi-directional streaming pull and the previous doesn't.\r\n\r\n**Test Setup:**\r\n- 100k messages published to a Topic\r\n- Single Subscription\r\n- _n_ Subscribers (k8s running on GKE, each pod running in separate node) {_n_ ∈ Z |  1 ≤ _n_ ≤ 4}\r\n- Each node had 2 vCPU and each Subscriber provided 10 user threads to the client library\r\n- All Subscribers running the same code (Parse message, build various classes, callback processing takes ~20ms)\r\n- Flow control settings: MaxOutstandingElementCount: 1000\r\n**Results:**\r\n- _n_ Messages remain stuck on Topic _n_  ∈ [223, 6652]\r\n**Observations:**\r\n- When messages are stuck: mod_ack_deadline_request_count  \u003e 0, Pull request count \u003e 0\r\n- When messages are stuck Subscriber callback threads are Waiting\r\n- No stuck messages when retesting with default Flow Control Settings (i.e. MaxOutstandingElementCount = null)\r\n- Experiments with MaxOutstandingElementCount ∈ [1000, 500, 250, 125] resulted in stuck message count ∈ [~6K, ~3K, ~1.7K, ~6K]\r\n\r\n**Interpretation:**\r\nMessage ACK deadlines are being continuously extended and the callback threads are not being invoked\r\n","author":{"url":"https://github.com/robertsaxby","@type":"Person","name":"robertsaxby"},"datePublished":"2017-09-19T12:02:54.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":8},"url":"https://github.com/2452/google-cloud-java/issues/2452"}