René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Migration to Py.test testing framework","articleBody":"This task has been started in PR #5405, but ended without success, so I have decided to split it to multiple stages:\r\n1. [x] Make `pytest` compatible with the current test suite\r\n   - [x] Add support of unitest-like fixtures (`setUp` and other methods) on classes not derived from `unitest.UnitTest` (fixed with https://github.com/matplotlib/matplotlib/pull/6730/commits/8b64294b667ac4d5a7b95c8b1cad22766db6de66)\r\n   - [x] Add support of `image_comparison` and `knownfailif` decorator (fixed with https://github.com/matplotlib/matplotlib/pull/6730/commits/bbaf7dc23fa92029fa9457e258a5ca4213c81851)\r\n   - [x] Fix `raise KnownFailureTest` problems (PR #6730)\r\n     - [x] `ImageComparisonTest.test`\r\n     - [x] `test_axes.test_formatter_large_small`\r\n     - [x] `test_backend_pgf.compare_figure`\r\n     - [x] `test_animation.check_save_animation`\r\n     - [x] `test_coding_standards.test_pep8_conformance_examples`\r\n   - [x] `test_basic` leaks tests from `numpy` into global scope with `from pylab import *` (fixed with https://github.com/matplotlib/matplotlib/pull/6730/commits/30caf00b60d8fe94c151d3d87670d355275d0568)\r\n   - [x] Solve strange failures on several tests\r\n     - [x] `test_subplots.test_subplots_offsettext` (PR #6741)\r\n     - [x] `test_axes.test_eventplot_problem_kwargs` (PR #6761)\r\n     - [x] `test_pickle.test_complete` (PR #6840)\r\n   - [x] Ensure that pytest collects all the tests (6150/6150) (actually nose shows 6151 because it counts  `lib/matplotlib/tests/test_delaunay.py::make_all_2d_testfuncs` as a test)\r\n2. [x] Migration to pytest\r\n   - [x] Add new build under `allow_failures` with pytest \r\n     - [x] Appveyor\r\n     - [x] Travis\r\n     - [x] Use `pytest-cov` for coverage reports\r\n   - [x] Replace nose in all builds with pytest\r\n3. [x] Post-migration\r\n   - [x] Rename `setUp` -\u003e `setup_method` and other methods to `pytest` analogs\r\n     - [x] Do not inherit tests from `unitest.UnitTest` class\r\n     - [x] Remove `setUp`/`tearDown`  workaround\r\n   - [x] Rewrite `image_comparison` decorator (currently it slowdowns the collection phase due to baseline images copying) or use `pytest-mpl` plugin (needs some improvements, see paragraph below)\r\n   - [x] Rewrite all assertations to pure `assert`\r\n   - [x] Get away from `cleanup` decorator\r\n   - [x] Replace `knownfailif` with `mark.xfail`\r\n   - [x] Replace `raise nose.SkipTest(...)` with `xfail(...)` call or `mark.skipif` decorator\r\n4. [ ] Enhancements\r\n   - [ ] Replace `setup_method` and etc methods with fixtures\r\n   - [ ] ~~Rewrite `test_delaunay.make_all_2d_testfuncs`~~\r\n   - [x] Use `pytest-pep8` plugin instead of `test_coding_standards`\r\n   - [ ] ~~Use `pytest-mock` plugin instead of `mock` and `unittest.mock`~~\r\n### Tests entry point (raised in https://github.com/matplotlib/matplotlib/pull/5405#issuecomment-154600278)\r\n\r\nI think that running tests with simple `py.test` command is better than `python tests.py`, but there are unused tests which will run that way. Solutions:\r\n- A. Get them away from pytest's sight with:\r\n  1. [ ] ~~Remove unused tests (`test_sankey`, `test_skew`, `test_ttconv`, and `test_usetex` from `lib/matplotlib/tests`)~~\r\n  2. [x] Rename `test_only.py` (to `setup_test_only.py` probably)\r\n- B. Make filter for collection phase.\r\n  1. [x] Implement whitelist\r\n  2. [x] Implement blacklist\r\n\r\nI have tried both ways and they work, so what to choose is up to you.\r\n- `py.test --collect-filter=none`\r\n- `py.test --collect-filter=blacklist`\r\n- `py.test --collect-filter=whitelist`\r\n\r\nCurrently `--collect-filter=whitelist` is default (to follow same politics as nose does currently).\r\n### Image comparison decorator\r\n\r\nWhy do we have to rewrite `image_comparison` decorator? Because `pytest-mpl` plugin in the current state (\"as is\") is not usable for matplotlib:\r\n- \\+ supports `savefig_kwargs`\r\n- \\+ supports `tolerance`\r\n- ‒ need to supply `baseline_dir` in every decorator\r\n- ‒ does not support image format conversion\r\n- ‒ need something to do with `freetype_version`, `remove_text`, and `style`\r\n","author":{"url":"https://github.com/Kojoley","@type":"Person","name":"Kojoley"},"datePublished":"2016-07-12T18:23:21.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":23},"url":"https://github.com/6731/matplotlib/issues/6731"}