René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Improve handling of \"Log entry with size X  exceeds maximum size\"","articleBody":"Hi,\r\n\r\nLib version: google-cloud-logging==1.15.0 \r\n\r\nI'm experiencing lots of errors like:\r\n```\r\nTraceback (most recent call last):\r\n  File \"/env/lib/python3.7/site-packages/google/cloud/logging/handlers/transports/background_thread.py\", line 123, in _safely_commit_batch\r\n    batch.commit()\r\n  File \"/env/lib/python3.7/site-packages/google/cloud/logging/logger.py\", line 383, in commit\r\n    client.logging_api.write_entries(entries, **kwargs)\r\n  File \"/env/lib/python3.7/site-packages/google/cloud/logging/_gapic.py\", line 126, in write_entries\r\n    partial_success=partial_success,\r\n  File \"/env/lib/python3.7/site-packages/google/cloud/logging_v2/gapic/logging_service_v2_client.py\", line 477, in write_log_entries\r\n    request, retry=retry, timeout=timeout, metadata=metadata\r\n  File \"/env/lib/python3.7/site-packages/google/api_core/gapic_v1/method.py\", line 143, in __call__\r\n    return wrapped_func(*args, **kwargs)\r\n  File \"/env/lib/python3.7/site-packages/google/api_core/retry.py\", line 286, in retry_wrapped_func\r\n    on_error=on_error,\r\n  File \"/env/lib/python3.7/site-packages/google/api_core/retry.py\", line 184, in retry_target\r\n    return target()\r\n  File \"/env/lib/python3.7/site-packages/google/api_core/timeout.py\", line 214, in func_with_timeout\r\n    return func(*args, **kwargs)\r\n  File \"/env/lib/python3.7/site-packages/google/api_core/grpc_helpers.py\", line 59, in error_remapped_callable\r\n    six.raise_from(exceptions.from_grpc_error(exc), exc)\r\n  File \"\u003cstring\u003e\", line 3, in raise_from\r\ngoogle.api_core.exceptions.InvalidArgument: 400 Log entry with size 256.6K exceeds maximum size of 256.0K\r\n```\r\nThis traceback doesn't help me identify place where the offending line was logged (I'm using python logging), so I cannot fix  it.\r\n\r\nI can see at least 3 resolutions to this:\r\n- allow user to specify that log message can be cropped to max size before sending to Cloud Logging. This way, if someone is ok with it, everything will get (at least partially) logged\r\n- add to the traceback information where the offending log entry was logged (file name + line number). This way I may be able to identify and fix too long log lines - right now there's no such information.\r\n- verify the log  message size eagerly when logging, i.e. throw exception from methods like `logging.info` when the message is too long and we won't be able to upload it (so rather than doing the verification in a background thread, do it immediately)\r\n\r\n","author":{"url":"https://github.com/przemekpastuszka","@type":"Person","name":"przemekpastuszka"},"datePublished":"2020-07-08T11:38:50.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":20},"url":"https://github.com/53/python-logging/issues/53"}