René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"BlobReader not buffering properly.","articleBody":"#### Environment details\r\n\r\n  - OS type and version: Mac 10.15.7\r\n  - Python version: `Python 3.9.5`\r\n  - pip version: `pip 21.1.1` \r\n  - `google-cloud-storage`: `Version: 1.38.0` \r\n\r\n#### Steps to reproduce\r\n\r\nWhen trying to stream a file from Google Cloud Storage to Google Drive, the BlobReader doesn't appear to be buffering properly.\r\n\r\nReading through the blobReader code, it should buffer the file as per chunksize, then download new chuncks as that buffer is exausted. However my experience is that every time the blobwriter is read a 2nd time, it invalidates the buffer, and downloads a new chunk.\r\n\r\nthe Google API `MediaIoBaseUpload` appears to be requesting files in `8192` bytes chunks, and every time the next chunk is requested from the GCS BlobReader, it downloads the next 40Mb chunk, rather than reading from the buffer.\r\n\r\nMy debugging has found that the buffer is actually being invalided when the python HTTP class 'seeks' the next chunk, and the math is failing [here](https://github.com/googleapis/python-storage/blob/master/google/cloud/storage/fileio.py#L155), however, i'm unsure what should be happening.\r\n\r\nCode example below demonstrates the problem, just provide your own client credentials for Drive, and upload a CSV file to Google Cloud storage, and note the blob and bucket.\r\n\r\n#### Code example\r\n\r\n```python\r\nfrom google.cloud import storage\r\nfrom googleapiclient.http import MediaIoBaseUpload\r\nfrom googleapiclient.discovery import build\r\nfrom google_auth_oauthlib.flow import InstalledAppFlow\r\nfrom google.auth.transport.requests import Request\r\nfrom google.oauth2.credentials import Credentials\r\nimport os\r\nimport logging\r\nimport sys\r\n\r\nlogging.basicConfig(stream=sys.stdout, level=logging.DEBUG)\r\n\r\nlogger = logging.getLogger(__name__)\r\n\r\nSCOPES = ['https://www.googleapis.com/auth/drive']\r\n\r\n\r\ndef getCreds():\r\n    creds = None\r\n    # The file token.json stores the user's access and refresh tokens, and is\r\n    # created automatically when the authorization flow completes for the first\r\n    # time.\r\n    if os.path.exists('token.json'):\r\n        creds = Credentials.from_authorized_user_file('token.json', SCOPES)\r\n    # If there are no (valid) credentials available, let the user log in.\r\n    if not creds or not creds.valid:\r\n        if creds and creds.expired and creds.refresh_token:\r\n            creds.refresh(Request())\r\n        else:\r\n            flow = InstalledAppFlow.from_client_secrets_file(\r\n                'client_secret.json', SCOPES)\r\n            creds = flow.run_local_server(port=0)\r\n        # Save the credentials for the next run\r\n        with open('token.json', 'w') as token:\r\n            token.write(creds.to_json())\r\n    \r\n    return creds\r\n\r\ndef gcsToDrive(bucketName, blobName):\r\n    client = storage.Client()\r\n\r\n    bucket = client.get_bucket(bucketName)\r\n    blob = bucket.blob(blobName)\r\n\r\n    creds = getCreds()\r\n\r\n    service = build('drive', 'v3', credentials=creds)\r\n\r\n    megabyte = (256 * 1024) * 4\r\n    chunk_size: int = megabyte * 40\r\n\r\n    with blob.open(\"rb\", chunk_size=chunk_size) as stream:\r\n        file_metadata = {\r\n            \"name\": \"My Report\",\r\n            \"mimeType\": \"application/vnd.google-apps.spreadsheet\",\r\n        }\r\n        media = MediaIoBaseUpload(stream, mimetype=\"text/csv\", resumable=True, chunksize=chunk_size)\r\n        file = (\r\n            service.files()\r\n            .create(body=file_metadata, media_body=media, fields=\"id\")\r\n            .execute()\r\n        )\r\n\r\n    logger.info(file)\r\n    \r\n    return file\r\n\r\nif __name__ == \"__main__\":\r\n    my_bucket = \"my_bucked\"\r\n    my_blob = \"my-blob.csv\"\r\n\r\n    gcsToDrive(my_bucket, my_blob)\r\n```\r\n\r\n#### Example Logs\r\n```\r\nDEBUG:google.auth._default:Checking /Users/\u003cusername\u003e/code/python/gscFileError/sa.json for explicit credentials as part of auth process...\r\nDEBUG:google.auth._default:Checking /Users/\u003cusername\u003e/code/python/gscFileError/sa.json for explicit credentials as part of auth process...\r\nDEBUG:urllib3.util.retry:Converted retries value: 3 -\u003e Retry(total=3, connect=None, read=None, redirect=None, status=None)\r\nDEBUG:google.auth.transport.requests:Making request: POST https://oauth2.googleapis.com/token\r\nDEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): oauth2.googleapis.com:443\r\nDEBUG:urllib3.connectionpool:https://oauth2.googleapis.com:443 \"POST /token HTTP/1.1\" 200 None\r\nDEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): storage.googleapis.com:443\r\nDEBUG:urllib3.connectionpool:https://storage.googleapis.com:443 \"GET /storage/v1/b/\u003cmy bucket\u003e?projection=noAcl\u0026prettyPrint=false HTTP/1.1\" 200 574\r\nINFO:googleapiclient.discovery_cache:file_cache is only supported with oauth2client\u003c4.0.0\r\nDEBUG:urllib3.connectionpool:https://storage.googleapis.com:443 \"GET /storage/v1/b/\u003cmy bucket\u003e/o/my-blob.csv?projection=noAcl\u0026prettyPrint=false HTTP/1.1\" 200 738\r\nDEBUG:googleapiclient.discovery:URL being requested: POST https://www.googleapis.com/upload/drive/v3/files?fields=id\u0026alt=json\u0026uploadType=resumable\r\nDEBUG:urllib3.connectionpool:https://storage.googleapis.com:443 \"GET /download/storage/v1/b/\u003cmy bucket\u003e/o/my-blob.csv?generation=1623219180016524\u0026alt=media HTTP/1.1\" 206 2240745\r\nDEBUG:urllib3.connectionpool:https://storage.googleapis.com:443 \"GET /download/storage/v1/b/\u003cmy bucket\u003e/o/my-blob.csv?generation=1623219180016524\u0026alt=media HTTP/1.1\" 206 2224361\r\nDEBUG:urllib3.connectionpool:https://storage.googleapis.com:443 \"GET /download/storage/v1/b/\u003cmy bucket\u003e/o/my-blob.csv?generation=1623219180016524\u0026alt=media HTTP/1.1\" 206 2207977\r\nDEBUG:urllib3.connectionpool:https://storage.googleapis.com:443 \"GET /download/storage/v1/b/\u003cmy bucket\u003e/o/my-blob.csv?generation=1623219180016524\u0026alt=media HTTP/1.1\" 206 2191593\r\nDEBUG:urllib3.connectionpool:https://storage.googleapis.com:443 \"GET /download/storage/v1/b/\u003cmy bucket\u003e/o/my-blob.csv?generation=1623219180016524\u0026alt=media HTTP/1.1\" 206 2175209\r\nDEBUG:urllib3.connectionpool:https://storage.googleapis.com:443 \"GET /download/storage/v1/b/\u003cmy bucket\u003e/o/my-blob.csv?generation=1623219180016524\u0026alt=media HTTP/1.1\" 206 2158825\r\nDEBUG:urllib3.connectionpool:https://storage.googleapis.com:443 \"GET /download/storage/v1/b/\u003cmy bucket\u003e/o/my-blob.csv?generation=1623219180016524\u0026alt=media HTTP/1.1\" 206 2142441\r\nDEBUG:urllib3.connectionpool:https://storage.googleapis.com:443 \"GET /download/storage/v1/b/\u003cmy bucket\u003e/o/my-blob.csv?generation=1623219180016524\u0026alt=media HTTP/1.1\" 206 2126057\r\n...\r\n...\r\n...\r\nDEBUG:urllib3.connectionpool:https://storage.googleapis.com:443 \"GET /download/storage/v1/b/\u003cmy bucket\u003e/o/my-blob.csv?generation=1623219180016524\u0026alt=media HTTP/1.1\" 206 94441\r\nDEBUG:urllib3.connectionpool:https://storage.googleapis.com:443 \"GET /download/storage/v1/b/\u003cmy bucket\u003e/o/my-blob.csv?generation=1623219180016524\u0026alt=media HTTP/1.1\" 206 78057\r\nDEBUG:urllib3.connectionpool:https://storage.googleapis.com:443 \"GET /download/storage/v1/b/\u003cmy bucket\u003e/o/my-blob.csv?generation=1623219180016524\u0026alt=media HTTP/1.1\" 206 61673\r\nDEBUG:urllib3.connectionpool:https://storage.googleapis.com:443 \"GET /download/storage/v1/b/\u003cmy bucket\u003e/o/my-blob.csv?generation=1623219180016524\u0026alt=media HTTP/1.1\" 206 45289\r\nDEBUG:urllib3.connectionpool:https://storage.googleapis.com:443 \"GET /download/storage/v1/b/\u003cmy bucket\u003e/o/my-blob.csv?generation=1623219180016524\u0026alt=media HTTP/1.1\" 206 28905\r\nDEBUG:urllib3.connectionpool:https://storage.googleapis.com:443 \"GET /download/storage/v1/b/\u003cmy bucket\u003e/o/my-blob.csv?generation=1623219180016524\u0026alt=media HTTP/1.1\" 206 12521\r\nDEBUG:urllib3.connectionpool:https://storage.googleapis.com:443 \"GET /download/storage/v1/b/\u003cmy bucket\u003e/o/my-blob.csv?generation=1623219180016524\u0026alt=media HTTP/1.1\" 416 29\r\nDEBUG:urllib3.connectionpool:https://storage.googleapis.com:443 \"GET /download/storage/v1/b/\u003cmy bucket\u003e/o/my-blob.csv?generation=1623219180016524\u0026alt=media HTTP/1.1\" 416 29\r\nINFO:__main__:{'id': '19GXqCAQDsRtK1czPSB_vX_SaBp4JtatlxCl4uQUU66o'}\r\n```\r\n","author":{"url":"https://github.com/Megabytemb","@type":"Person","name":"Megabytemb"},"datePublished":"2021-06-10T03:05:52.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":8},"url":"https://github.com/462/python-storage/issues/462"}