René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Request batching does not track encoding overhead accurately","articleBody":"#### Steps to reproduce\r\n\r\n  1. Set the byte size per batched request to \u003e9.1 MB.\r\n  2. Construct a batch of sufficiently large messages while respecting limits per message (max 10 MB data + attributes + ordering key, max 100 attributes, max 256 B per attribute key, max 1024 B per attribute value, max 1024 B per ordering key) to exceed either the serialized size limit of 10 MB per `PublishRequest` or 10 MiB request size limit per call to publish (pointless because of the first limit unless explicit size validation is enabled).\r\n  3. Publish the batch.\r\n  4. Observe thrown exception, either an early rejection (\u003e10 MiB) or an internal error (\u003e10 MB, \u003c10 MiB) on request_size if explicit size validation isn't enabled for the topic.\r\n\r\n#### External references such as API reference guides\r\n\r\nThe API reference does not provide adequate documentation on these limits, since it suggests that explicit size validation is used. \r\n\r\n#### Any additional information below\r\n\r\nDiscovered this while reviewing size validation in `PubsubIO` for Apache Beam, see apache/beam#31800 for details.\r\nHappy to open a PR with fixes when I get a chance, just wanted to highlight this ahead of time for discussion.\r\n\r\nThis shouldn't be an issue if a user recovers by lowering the byte size per batch (\u003c9 MB is generally safe with current limits per message), but the client library should track the outgoing request size accurately on the user's behalf in my opinion.\r\n\r\nAlso note that allowing users to use the REST API for Pub/Sub as proposed in #2075 would complicate matters since the serialized size of publish requests sent as JSON can be smaller than the serialized size of the same requests after it's transcoded into protobuf. For this message type a protobuf message can be up to ~38 KB smaller or up to ~0.9 MB larger than its JSON counterpart.\r\nAn example of this would be a JSON publish request of 736 messages, each message with no data and 100 attributes, each attribute with no value and a key of 128 alphanumeric ASCII characters. The JSON request does not exceed 10 MB, but the protobuf transcoded request that's passed to Pub/Sub exceeds it by ~145 KB. That triggers the internal validation error without explicit size validation, but I guess this will become a non-issue when explicit size validation is used by default.","author":{"url":"https://github.com/sjvanrossum","@type":"Person","name":"sjvanrossum"},"datePublished":"2024-07-08T16:44:42.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":2},"url":"https://github.com/2106/java-pubsub/issues/2106"}