René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Make Cloud Storage client retry on backend error","articleBody":"We're operating at scale on GCS and are regularly experiencing transient [HTTP 410 status codes](https://cloud.google.com/storage/docs/json_api/v1/status-codes#410_Gone) when accessing Cloud storage. Those 410 status codes returned by Cloud storage are bogus though, as they are effectively just hiding an internal backend error on GCS, which is reflected in the error details:\r\n\r\n```\r\nCaused by: com.google.api.client.googleapis.json.GoogleJsonResponseException: \r\n410 Gone { \"code\" : 503, \"errors\" : [ { \"domain\" : \"global\", \"message\" : \"Backend Error\", \"reason\" : \"backendError\" } ], \"message\" : \"Backend Error\" }\r\n```\r\n\r\nThe google-cloud-storage client does not treat the 410 status code as retryable, understandibly so. It should be retrying on backend errors, though, which are typically exposed with status code 500 or 503. **I'm suggesting to treat backend errors in the client [in the same way as it treats internal errors](https://github.com/GoogleCloudPlatform/google-cloud-java/blob/master/google-cloud-clients/google-cloud-storage/src/main/java/com/google/cloud/storage/StorageException.java#L45), namely match on `reason == backendError` independently of HTTP status code.**\r\n\r\nNote that we're not the first ones to experience this, and the client should be resilient against these transient GCS errors.\r\n- https://stackoverflow.com/questions/33056415/uploading-files-into-google-cloud-storage-500-backend-error\r\n- https://stackoverflow.com/questions/35125891/google-cloud-dataflow-jobs-failing-inaccessible-jars-410-gone-errors\r\n- https://stackoverflow.com/questions/41215541/dataflow-jobs-fail-after-a-few-410-errors-while-writing-to-gcs","author":{"url":"https://github.com/ywelsch","@type":"Person","name":"ywelsch"},"datePublished":"2018-08-21T08:23:39.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":7},"url":"https://github.com/3586/google-cloud-java/issues/3586"}