René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"PostgresOnlineStore: Improve materialization","articleBody":"**Is your feature request related to a problem? Please describe.**\r\nThe current implementation of `PostgreSQLOnlineStoreConfig.online_write_batch()` is quite slow. \r\n\r\n**Describe the solution you'd like**\r\nOne of the suggested options discussed [in this issue](https://github.com/feast-dev/feast/issues/4036) is using [bulk writes](https://github.com/apache/arrow-adbc/pull/1093) with [adbc](https://arrow.apache.org/adbc/main/driver/postgresql.html). \r\n\r\n**Describe alternatives you've considered**\r\nAnother option could be to use `COPY FROM` command. This [performance benchmark](https://www.timescale.com/blog/psycopg2-vs-psycopg3-performance-benchmark/) shows that the number of rows written per second is significantly larger compared to the batch insert we are doing now. \r\n\r\nThis image shows the comparison from the performance benchmark: \r\n\u003cimg width=\"607\" alt=\"Screenshot 2024-06-24 at 11 48 25\" src=\"https://github.com/feast-dev/feast/assets/55230856/c5976c8c-93c2-484a-89b1-8fd42acfbf9a\"\u003e\r\n\r\n**Additional context**\r\nI think we should benchmark both solutions and pick the one that achieves the best performance.\r\n","author":{"url":"https://github.com/job-almekinders","@type":"Person","name":"job-almekinders"},"datePublished":"2024-06-24T09:34:37.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":18},"url":"https://github.com/4309/feast/issues/4309"}