René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Question: How to optimize load_data-Operation","articleBody":"I want to copy my MySQL data (\u003e200 Mio. rows) to BigQuery. Therefore I created a python script, which uses this library. At the moment it streams 1000 rows with one request and it generates about 1,1 requests/second. This is not really fast and it would take me days to transfer the whole dataset. I am sure that this can be optimized, but I don't know how. Would you have some suggestions? You can find my source code [here](https://github.com/inkrement/MySQLbq/blob/e8f484a5aac93dc77f687457424995a72ad4460b/run.py)\r\n\r\nI thought about the following points:\r\n\r\n * Each request contains 1000 rows, should I choose a bigger number?\r\n * Does this library use gzip per default?","author":{"url":"https://github.com/inkrement","@type":"Person","name":"inkrement"},"datePublished":"2017-01-23T21:51:37.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":11},"url":"https://github.com/2960/google-cloud-python/issues/2960"}