René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Proposal for the baseline images problem","articleBody":"This is a proposal to help with the difficulty in adding/modifying tests which require a baseline image.  As a brief reminder, baseline images are problematic because\r\n- they cause the repo size to grow rather quickly (matplotlib's .git is now ~400Mb, compared e.g. to numpy's which is 72Mb), even though we've been exceedingly careful in recent years to avoid adding more baseline images (the problems with the merge of #8896 got me to think about this again).\r\n- they force us to pin to a somewhat old version of freetype, because nearly *every* release of freetype causes tiny rasterization changes that would entail regenerating all baseline images (and thus cause even more repo size growth).\r\n\r\nThe idea is to actually *not* store baseline images in the repo at all.\r\n1) When generating a sdist or wheels for a new release, we do generate baseline images (from the tests) and store them in the released sdist or wheel.  The validity of the baseline images is guaranteed by what comes below.\r\n2) During normal development, the baseline images should not change.  If someone wants to run the test suite from a git checkout, matplotlib would first generate a set of \"known\" baseline images either by grabbing the last release, or just installing the last tagged release into a temporary venv and running the test suite there (of course, they would be cached into ~/.cache/matplotlib so this is only done once -- this caching also needs to be done so that CI time doesn't double...).\r\n3) If a baseline image needs to change (we should still try to keep this as rare as possible), the *commit message* needs to include a special string e.g. `[baseline-change]: path/to/image`.  The process described in 2) would also consult the git log to know if there's any images that need to be gen'd in such a way.  Likewise, if a baseline image needs to be added for a new test, the commit message needs to be tagged accordingly.  Possibly, we could also require that the filename also be changed in that case so that even if you're not in a git repo, you at least don't end up using the wrong baseline image, and instead just get a skip.\r\n\r\nNote that this means that if you're using a non-git-repo, non-sdist/wheel, you cannot run the full test suite anymore, because you at least need to know what was the last release, and if any image changed since then, you'd like to know when (however, if you want to write a patch and check that it's good, this still works because you can compare the baselines with what was present before your patch).  OTOH distro/conda packagers, which *should* be working from sdists, would still be able to run the test suite fine, because the sdists/wheels will include baseline images per 1).\r\n\r\nThere's a bit of machinery involved in making this work, but I don't think it's overly hard to implement.  Thoughts?","author":{"url":"https://github.com/anntzer","@type":"Person","name":"anntzer"},"datePublished":"2020-02-08T23:14:41.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":98},"url":"https://github.com/16447/matplotlib/issues/16447"}