René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Discussion: Pushing batches of data to online store: Should `conn.commit()` happen in the for loop or after?","articleBody":"[This](https://github.com/feast-dev/feast/blob/master/sdk/python/feast/infra/online_stores/contrib/postgres.py#L80-L101) piece of code in the `online_write_batch` function in the Postgres online store pushes data in batches to the online store. \r\n\r\nI was wondering whether it makes more sense to put the `conn.commit()` inside the for loop, or after the for loop. I would love to hear the different trade-offs between the two options! \r\n\r\nCopy of the code snippet here: \r\n\r\n```python\r\nbatch_size = 5000\r\nfor i in range(0, len(insert_values), batch_size):\r\n    cur_batch = insert_values[i : i + batch_size]\r\n    execute_values(\r\n        cur,\r\n        sql.SQL(\r\n            \"\"\"\r\n            INSERT INTO {}\r\n            (entity_key, feature_name, value, event_ts, created_ts)\r\n            VALUES %s\r\n            ON CONFLICT (entity_key, feature_name) DO\r\n            UPDATE SET\r\n                value = EXCLUDED.value,\r\n                event_ts = EXCLUDED.event_ts,\r\n                created_ts = EXCLUDED.created_ts;\r\n            \"\"\",\r\n        ).format(sql.Identifier(_table_id(project, table))),\r\n        cur_batch,\r\n        page_size=batch_size,\r\n    )\r\n    conn.commit() # \u003c\u003c This is the point of interest. Should this be placed in the loop or after?  \r\n```","author":{"url":"https://github.com/job-almekinders","@type":"Person","name":"job-almekinders"},"datePublished":"2024-03-23T10:55:18.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":4},"url":"https://github.com/4036/feast/issues/4036"}