René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"CI: re-consider how we test GUI memory leaks","articleBody":"https://github.com/matplotlib/matplotlib/pull/22002 fixed some (but not all) memory leaks it tkagg as well as added tests to put an upper bound on the amount of memory that lost when making GUI windows in a tight Python loop and only periodically running the event loop.\r\n \r\nHowever after merging we found that these tests were found that the memory consumed had significant variability on OSX on both the OSX and tkagg backends (bug report: #22959 patches: #22987 , #22961 ).  While pragmatic, bumping the threshold is giving a reminder of when our image comparisons had some tolerance in them and we ended up using the wrong glyph for a while even though we had an explicit test (it passed with in tolerance!).  It is possible I have over-learned this lesson and should chill.\r\n\r\nWhen talking about this on the dev call this week we had at least one wrong idea about the source of the variability and we still do not understand it.  Due to the variability it is easily possible that a small leak will not be robustly detectable under the noise and hence the test would not actually test what we want to test.  We could get around that my running the tests for much longer (like we do in https://github.com/matplotlib/matplotlib/blob/main/tools/memleak.py) , however that would result in a test that runs for 10s or greater which is very much not ideal for a test suite (either localy or on CI). \r\n\r\nTo this end we had a couple of ideas:\r\n - push the memory leakage testing into its own CI job (and only run it on merge / on a clock)?\r\n - use ASV to run these tests\r\n   - maybe on macstadium, maybe on ARM test box we have, maybe on digital ocean\r\n\r\nIn the very short term I think we should\r\n1. merge #22987 \r\n2. if 1. fails, skip these tests on osx\r\n3. If we think 2. too much just hiding the bad case under the carpet, pull these tests out and get one of the above ideas done ASAP.","author":{"url":"https://github.com/tacaswell","@type":"Person","name":"tacaswell"},"datePublished":"2022-05-06T02:09:10.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":1},"url":"https://github.com/22988/matplotlib/issues/22988"}