René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Include test files in coverage report","articleBody":"Of course I can run coverage for tests locally, but I think it should be accessible on coveralls. You cannot blindly rely on a test report until you are sure that they are actually working, otherwise it could give you a false sense of security.\n\nI have faced a problem where I cannot be sure that pytest runs all the tests. Moreover I have got a test failures with pytest on tests that nose does not perform (and I do not why then they are not deleted). Here is the list:\n- [test_sankey](https://coveralls.io/builds/7303404/source?filename=lib%2Fmatplotlib%2Ftests%2Ftest_sankey.py) not deprecated, runs without issues\n- [test_skew](https://coveralls.io/builds/7303404/source?filename=lib%2Fmatplotlib%2Ftests%2Ftest_skew.py) not deprecated, has outdated baseline images or tolerance problems (`..FF.F`: RMS 0.014, 0.264, 0.065)\n- [test_ttconv](https://coveralls.io/builds/7303404/source?filename=lib%2Fmatplotlib%2Ftests%2Ftest_ttconv.py) not deprecated, runs without issues\n- [test_usetex](https://coveralls.io/builds/7303404/source?filename=lib%2Fmatplotlib%2Ftests%2Ftest_usetex.py) not deprecated, has outdated baseline images or tolerance problems (`.F`: RMS 0.261)\n\nWhat benefits can we have by including test files coverage in the report? I collected dark spots in the test suite:\n\nImportant (test not running, uncovered cases):\n- [test_animation:test_no_length_frames](https://coveralls.io/builds/7303404/source?filename=lib%2Fmatplotlib%2Ftests%2Ftest_animation.py#L150)\n- [test_axes:test_pcolorargs](https://coveralls.io/builds/7303404/source?filename=lib%2Fmatplotlib%2Ftests%2Ftest_axes.py#L763)\n- [test_backend_pdf:test_use14corefonts](https://coveralls.io/builds/7303404/source?filename=lib%2Fmatplotlib%2Ftests%2Ftest_backend_pdf.py#L18) does not run on travis (`TRAVIS` env variable was removed?)\n- [test_contour:test_contour_manual_labels](https://coveralls.io/builds/7303404/source?filename=lib%2Fmatplotlib%2Ftests%2Ftest_contour.py#L157)\n- [test_image:test_minimized_rasterized](https://coveralls.io/builds/7303404/source?filename=lib%2Fmatplotlib%2Ftests%2Ftest_image.py#L567)\n- [test_mlab:Test_detrend.test_detrend_mean_2D_none](https://coveralls.io/builds/7303404/source?filename=lib%2Fmatplotlib%2Ftests%2Ftest_mlab.py#L878)\n- [test_mlab:Test_detrend.test_detrend_mean_2D_none_T](https://coveralls.io/builds/7303404/source?filename=lib%2Fmatplotlib%2Ftests%2Ftest_mlab.py#L889)\n- [test_mlab:Test_detrend.test_detrend_mean_2D_axis0](https://coveralls.io/builds/7303404/source?filename=lib%2Fmatplotlib%2Ftests%2Ftest_mlab.py#L900)\n- [test_mlab:Test_detrend.test_detrend_mean_2D_axis1](https://coveralls.io/builds/7303404/source?filename=lib%2Fmatplotlib%2Ftests%2Ftest_mlab.py#L915)\n- [test_mlab:Test_detrend.test_detrend_mean_2D_axism1](https://coveralls.io/builds/7303404/source?filename=lib%2Fmatplotlib%2Ftests%2Ftest_mlab.py#L930)\n- [test_mlab:Test_detrend.test_detrend_str_linear_2d_slope_off_axis0](https://coveralls.io/builds/7303404/source?filename=lib%2Fmatplotlib%2Ftests%2Ftest_mlab.py#L1263)\n- [test_mlab:Test_detrend.test_detrend_detrend_linear_1d_slope_off_axis1](https://coveralls.io/builds/7303404/source?filename=lib%2Fmatplotlib%2Ftests%2Ftest_mlab.py#L1275)\n- [test_mlab:test_griddata_nn](https://coveralls.io/builds/7303404/source?filename=lib%2Fmatplotlib%2Ftests%2Ftest_mlab.py#L2817)\n- [test_subplots:test_shared](https://coveralls.io/builds/7303404/source?filename=lib%2Fmatplotlib%2Ftests%2Ftest_subplots.py#L15)\n- [test_patheffects:test_PathEffect_points_to_pixels](https://coveralls.io/builds/7303404/source?filename=lib%2Fmatplotlib%2Ftests%2Ftest_patheffects.py#L89)\n\nLess important (partial coverage):\n- [test_backend_svg _test_determinism](https://coveralls.io/builds/7303404/source?filename=lib%2Fmatplotlib%2Ftests%2Ftest_backend_svg.py#L126)\n- [test_cbook:test_results_whiskers_range](https://coveralls.io/builds/7303404/source?filename=lib%2Fmatplotlib%2Ftests%2Ftest_cbook.py#L227)\n- [test_cbook:test_results_whiskers_percentiles](https://coveralls.io/builds/7303404/source?filename=lib%2Fmatplotlib%2Ftests%2Ftest_cbook.py#L241)\n- [test_dates:test_date2num_dst](https://coveralls.io/builds/7303404/source?filename=lib%2Fmatplotlib%2Ftests%2Ftest_dates.py#L397)\n- ~~[test_delaunay](https://coveralls.io/builds/7303404/source?filename=lib%2Fmatplotlib%2Ftests%2Ftest_delaunay.py#L121)~~\n- [test_mlab:Test_window](https://coveralls.io/builds/7303404/source?filename=lib%2Fmatplotlib%2Ftests%2Ftest_mlab.py#L431)\n- [test_mlab:Test_spectral_nosig_real_onesided](https://coveralls.io/builds/7303404/source?filename=lib%2Fmatplotlib%2Ftests%2Ftest_mlab.py#L1338)\n- [test_units:test_numpy_facade](https://coveralls.io/builds/7303404/source?filename=lib%2Fmatplotlib%2Ftests%2Ftest_units.py#L36)\n\nNot important (requires tests for the tests, and other cases):\n- [test_collections check_allprop](https://coveralls.io/builds/7303404/source?filename=lib%2Fmatplotlib%2Ftests%2Ftest_collections.py#L395) unused function?\n- [test_collections check_segments](https://coveralls.io/builds/7303404/source?filename=lib%2Fmatplotlib%2Ftests%2Ftest_collections.py#L385)\n- [test_transforms:test_non_affine_caching transform_path_non_affine\u0026transform_non_affine](https://coveralls.io/builds/7303404/source?filename=lib%2Fmatplotlib%2Ftests%2Ftest_transforms.py#L42)\n- [test_triplot_return](https://coveralls.io/builds/7303404/source?filename=lib%2Fmatplotlib%2Ftests%2Ftest_triangulation.py#L957)\n\nSomeone can say that we need tests for the tests to have full coverage -- I would say no, some special places can be marked with `#pragma: no cover`, but other can be fixed to have full coverage.\n\nCurrently this can be done with `nose-cov` plugin and after migrating to pytest we will have exactly same coverage reports with `pytest-cov`.\n","author":{"url":"https://github.com/Kojoley","@type":"Person","name":"Kojoley"},"datePublished":"2016-08-04T17:03:24.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":6},"url":"https://github.com/6902/matplotlib/issues/6902"}