René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Fix old SCTs that fail assertions","articleBody":"Since `pythonwhat v2.16.0`, different SCT functions do some assertions and verifications to figure out whether the function is appropriately used. You can see it in the CHANGELOG [here](https://github.com/datacamp/pythonwhat/blob/master/CHANGELOG.md#2160).\r\n\r\nSome of these assertions are only done for courses whose course image contains the `PYTHONWHAT_V2_ONLY` env variable (more recent courses), because otherwise these changes would break 55 exercises that contain SCTs that violate these rules:\r\n\r\n```\r\n   course_id chapter_id     id number                                                                              title\r\n1        998       2244  18929      3                                                             Loading a pickled file\r\n2        998       2284  18950      8                                  Filtering your database records using SQL's WHERE\r\n3        998       2284  18951     14                  The power of SQL lies in relationships between tables: INNER JOIN\r\n4        998       2284  19468      7                                         Customizing the Hello World of SQL Queries\r\n5        998       2284  19469      9                                            Ordering your SQL records with ORDER BY\r\n6        998       2284  19793     11                                         Pandas and The Hello World of SQL Queries!\r\n7        998       2372  20216     12            Turning a webpage into data using BeautifulSoup: getting the hyperlinks\r\n8        998       2388  20437     13                                                 Load and explore your Twitter data\r\n9        998       2388  20441     15                                              A little bit of Twitter text analysis\r\n10       998       2388  20834      3                                                       Loading and exploring a JSON\r\n11      1115       2469  21424      3                                         Filter data selected from a Table - Simple\r\n12      1115       2470  21436      3                                       Calculating a Difference between Two Columns\r\n13      1115       2470  21442     10                                    Using alias to handle same table joined queries\r\n14      1115       2471  21450      7                                                         Loading a CSV into a Table\r\n15      1115       2471  21452      9                                                        Updating individual records\r\n16      1115       2472  21463      5                                                      Reading the Data from the CSV\r\n17      1115       2472  21466      8                           Build a Query to Determine the Average Age by Population\r\n18      1115       2472  21467      9        Build a Query to Determine the Percentage of Population by Gender and State\r\n19      1115       2472  21468     10 Build a Query to Determine the Difference by State from the 2000 and 2008 Censuses\r\n20      1115       2470  33053     11                          Leveraging Functions and Group_bys with Hierarchical Data\r\n21      1531       3844  38857      8                                                                          Using zip\r\n22      1531       3845  38874      9                                                                Dict comprehensions\r\n23      1550       3936  40011      8                                                                 How is it optimal?\r\n24      1550       3939  40032      8                                             Hypothesis test on Pearson correlation\r\n25      1550       3940  40049     17                                     Is beak depth heritable at all in G. scandens?\r\n26      1550       3937  40360      4                                                      Visualizing bootstrap samples\r\n27      1550       3938  47093      3                                                   Visualizing permutation sampling\r\n28      1550       3936  63252     12                                             Linear regression on all Anscombe data\r\n29      1606       4135  42717     12            Turning a webpage into data using BeautifulSoup: getting the hyperlinks\r\n30      1606       4136  42720      3                                                       Loading and exploring a JSON\r\n31      1606       4140  42795      6                                              A little bit of Twitter text analysis\r\n32      1607       4138  42757      3                                                             Loading a pickled file\r\n33      1607       4139  42780      8                                         Customizing the Hello World of SQL Queries\r\n34      1607       4139  42781      9                                  Filtering your database records using SQL's WHERE\r\n35      1607       4139  42782     10                                            Ordering your SQL records with ORDER BY\r\n36      1639       4284  59225     10                                                                    Sunny or cloudy\r\n37      1681       4403  62438     10                                  Concatenating vertically to get MultiIndexed rows\r\n38      1681       4403  62441     13                                               Concatenating DataFrames from a dict\r\n39      1681       4403  92426      8                                        Reading multiple files to build a DataFrame\r\n40      1822       4794  51256      4                                                             Finding open triangles\r\n41      2072       5698  63534      6                                                        How many clusters of grain?\r\n42      2533       7496  86163      9                                         Encode the labels as categorical variables\r\n43      2533       7496  95226     10                                                             Counting unique labels\r\n44      3629      10390 119216      9                                                       Regex with NLTK tokenization\r\n45      3629      10390 119217     10                                                             Non-ascii tokenization\r\n46      3629      10390 119219     12                                                                  Charting practice\r\n47      3679      10556 121072      8                                                    Using regularization in XGBoost\r\n48      3679      10557 121076      3                                               Tuning the number of boosting rounds\r\n49      3679      10557 227050      6                                                                         Tuning eta\r\n50      3679      10557 227051      7                                                                   Tuning max_depth\r\n51      3679      10557 227052      8                                                            Tuning colsample_bytree\r\n52      4299      13274 151880      8                                                              Filtering on a phrase\r\n53      6221      19972 258084     11                                                                  Balancing classes\r\n54      6221      19972 258085     12                                            Comparison of Employee attrition models\r\n55      7032      27285 321296     14                                                                   Detect edges (2)\r\n```\r\n\r\nThis issue is a reminder that we have to rewrite the SCTs for these exercises because they are smelly. As soon as that is done, we can remove this `v2_only()` check and do the assertions for all SCTs, independent of the `PYTHONWHAT_V2_ONLY` env variable.\r\n\r\n### How I came up with this list\r\n\r\n- Make a branch on `pythonwhat` that does not do the `if v2_only()` check\r\n- Install this branch of pythonwhat in `docker-python-shared` (using `pip install git+...`) and tag it (with a release candidate format, as it is never the intention to deploy)\r\n- This makes the validator go through all exercises with this hypothetical shared image, and spit out the exercises where it goes wrong.\r\n- Finally, you can use [this script](https://github.com/datacamp/content-engineering-scripts/blob/master/analysis_incorrect-scts.R) to programmatically a list of exercise ids from the validator endpoint and their corresponding chapters and courses.","author":{"url":"https://github.com/filipsch","@type":"Person","name":"filipsch"},"datePublished":"2018-09-12T11:22:31.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":2},"url":"https://github.com/351/pythonwhat/issues/351"}