René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Large Memory and Speed performance regression from 2.5.3 to 2.6.3","articleBody":"We upgraded to rails 7.1.3 and at the same time I upgraded our version of creek from 2.5.3 to 2.6.3.\r\n\r\nAfter deploying the new rails version, we noticed our memory usage going up significantly as well as our time to process spreadsheets.\r\n\r\n~~I tested the two versions in irb isolation and saw no difference, with a slight improvement for 2.6.3~~. However, when in docker the difference is massive.\r\n\r\n(See https://github.com/pythonicrubyist/creek/issues/122#issuecomment-1957764405, my testing methodology wasn't great for local Mac and the difference is less but still large. This is not just docker)\r\n\r\n```ruby\r\n[3] [app][development] pry(main)\u003e Creek::VERSION\r\n=\u003e \"2.5.3\"\r\n[4] [app][development] pry(main)\u003e time = Time.now; creek = Creek::Book.new('performance-test.xlsx'); puts(Time.now - time);\r\n8.105815712\r\n[5] [app][development] pry(main)\u003e time = Time.now; creek.sheets[0].simple_rows.map(\u0026:inspect); puts(Time.now - time);\r\n38.974233879\r\n[6] [app][development] pry(main)\u003e\r\n```\r\n\r\n```\r\nVmPeak:\t  709908 kB\r\nVmSize:\t  593128 kB\r\nVmLck:\t       0 kB\r\nVmPin:\t       0 kB\r\nVmHWM:\t  616644 kB\r\nVmRSS:\t  518568 kB\r\nVmData:\t  559256 kB\r\nVmStk:\t    8188 kB\r\nVmExe:\t       4 kB\r\nVmLib:\t   24516 kB\r\nVmPTE:\t    1372 kB\r\nVmSwap:\t       0 kB\r\n```\r\n\r\n```ruby\r\n[1] [app][development] pry(main)\u003e Creek::VERSION\r\n=\u003e \"2.6.3\"\r\n[2] [app][development] pry(main)\u003e time = Time.now; creek = Creek::Book.new('performance-test.xlsx'); puts(Time.now - time);\r\n8.186497337\r\n[3] [app][development] pry(main)\u003e time = Time.now; creek.sheets[0].simple_rows.map(\u0026:inspect); puts(Time.now - time);\r\n66.137462044\r\n[4] [app][development] pry(main)\u003e\r\n```\r\n\r\n```\r\nVmPeak:\t 3852528 kB\r\nVmSize:\t  603644 kB\r\nVmLck:\t       0 kB\r\nVmPin:\t       0 kB\r\nVmHWM:\t 3765544 kB\r\nVmRSS:\t  529308 kB\r\nVmData:\t  569772 kB\r\nVmStk:\t    8188 kB\r\nVmExe:\t       4 kB\r\nVmLib:\t   24516 kB\r\nVmPTE:\t    1520 kB\r\nVmSwap:\t       0 kB\r\n```\r\n\r\nAs you can see in the above, the problem appears to be the row generator. Also, the peak memory usage went from 709,000 KB to 3,852,000 KB, 5 times more memory!\r\n\r\nHere is the spreadsheet I used for testing (a subset of chicago's public crime data) https://drive.usercontent.google.com/download?id=1zWiCRYCS7Vs9EPZyOqcKr-eT59F7w3dm\u0026export=download (give it a minute to start downloading, google has to virus scan it first and it takes forever)\r\n","author":{"url":"https://github.com/azrazalea-debtbook","@type":"Person","name":"azrazalea-debtbook"},"datePublished":"2024-02-21T03:21:05.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":8},"url":"https://github.com/122/creek/issues/122"}