René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Add Data from wind-turbine-models.com","articleBody":"Hello everybody,\r\n\r\nI wrote a small script to parse the data from wind-turbine-models.com.\r\nAs the Turbine-data already contains records from wind-turbine-models.com (https://github.com/wind-python/windpowerlib/blob/master/windpowerlib/oedb/turbine_data.csv) I hope that the legal issues discussed in https://github.com/OpenEnergyPlatform/data-preprocessing/issues/28#issuecomment-808275889 are cleared and the data can be used.\r\n\r\nIt would be very good if the power curves could be additionally integrated into the OEP-Database.\r\n\r\nThe below code is available under the MIT License and free for anyone to use:\r\n```\r\nfrom bs4 import BeautifulSoup # parse html\r\nimport requests\r\nimport json5 # parse js-dict to python\r\nimport json\r\nimport pandas as pd\r\nfrom tqdm import tqdm # fancy for loop\r\n\r\n# create list of turbines with available powercurves\r\npage = requests.get('https://www.wind-turbine-models.com/powercurves')\r\nsoup = BeautifulSoup(page.text, 'html.parser')\r\n# pull all text from the div\r\nname_list = soup.find(class_ ='chosen-select')\r\n\r\nwind_turbines_with_curve = []\r\nfor i in name_list.find_all('option'):    \r\n    wind_turbines_with_curve.append(i.get('value'))\r\n\r\ndef downloadTurbineCurve(turbine_id,start = 0, stop=25):\r\n    url = \"https://www.wind-turbine-models.com/powercurves\"\r\n    headers = dict()\r\n    headers[\"Content-Type\"] = \"application/x-www-form-urlencoded\"\r\n    data = {'_action': 'compare', 'turbines[]': turbine_id, 'windrange[]': [start, stop]}\r\n\r\n    resp = requests.post(url, headers=headers, data=data)\r\n    strings = resp.json()['result']\r\n    begin = strings.find('data:')\r\n    end = strings.find('\"}]', begin)\r\n    relevant_js = '{'+strings[begin:end+3]+'}}'\r\n    curve_as_dict = json5.loads(relevant_js)\r\n    x = curve_as_dict['data']['labels']\r\n    y = curve_as_dict['data']['datasets'][0]['data']\r\n    label = curve_as_dict['data']['datasets'][0]['label']\r\n    url = curve_as_dict['data']['datasets'][0]['url']\r\n    df = pd.DataFrame(y, index=x, columns=[label])\r\n    df.index.name = 'wind_speed'\r\n    return df\r\n\r\ncurves = []\r\nfor turbine_id in tqdm(wind_turbines_with_curve):\r\n    curve = downloadTurbineCurve(turbine_id)\r\n    curves.append(curve)\r\nc = pd.concat(curves,axis=1)\r\nd = c[c.any(axis=1)]\r\n    \r\nwith open('down.csv','w') as f: \r\n    d.to_csv(f)\r\n```","author":{"url":"https://github.com/maurerle","@type":"Person","name":"maurerle"},"datePublished":"2021-11-26T15:50:11.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":1},"url":"https://github.com/120/windpowerlib/issues/120"}