René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"`Client.insert_rows_json()`: add option to disable best-effort deduplication","articleBody":"Currently, the `Client.insert_rows_json()` method for streaming inserts always inserts an `insertId` unique identifier for each row provided.\r\nThis row identifier can be user-provided; if the user doesn't provide any identifiers, the library automatically fills the row IDs by using UUID4.\r\n\r\nHere's the code:\r\n```python\r\n        for index, row in enumerate(json_rows):\r\n            info = {\"json\": row}\r\n            if row_ids is not None:\r\n                info[\"insertId\"] = row_ids[index]\r\n            else:\r\n                info[\"insertId\"] = str(uuid.uuid4())\r\n            rows_info.append(info)\r\n```\r\n\r\nHowever, insert IDs are entirely optional, and there are actually valid use cases _not_ to use them. From the [BigQuery documentation](https://cloud.google.com/bigquery/streaming-data-into-bigquery#disabling_best_effort_de-duplication):\r\n\r\n\u003e You can disable best effort de-duplication by not populating the insertId field for each row inserted. When you do not populate insertId, you get higher streaming ingest quotas in certain regions. This is the recommended way to get higher streaming ingest quota limits.\r\n\r\nThe BigQuery Python client library provides no way of omitting the `insertId`s. it would be nice to have a parameter for that.","author":{"url":"https://github.com/pietrodn","@type":"Person","name":"pietrodn"},"datePublished":"2021-06-25T15:08:21.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":6},"url":"https://github.com/720/python-bigquery/issues/720"}