René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Bug - cannot convert float NaN to integer in merge_hapi","articleBody":"merge_hapi() throws an error when the resulting dataframe contains a column of ints and NaNs because Python does not allow converting NaN to int.\r\n\r\nReproduce error:\r\n```python\r\nfrom hapiclient import hapi\r\nfrom hapiplot import hapiplot\r\nimport hapitools\r\n\r\nopts = {'logging': False, 'usecache': True, 'cachedir': './hapicache' }\r\nstart = '2013-01-01T00:00:54Z'\r\nstop = '2013-01-01T06:00:54.000Z'\r\nserverA, datasetA, parametersA = 'https://cdaweb.gsfc.nasa.gov/hapi', 'OMNI2_H0_MRG1HR', 'DST1800'\r\nserverB, datasetB, parametersB = \"https://imag-data.bgs.ac.uk/GIN_V1/hapi\", \"cki/best-avail/PT1M/hdzf\", \"Field_Vector\"\r\n\r\ndataA, metaA = hapi(serverA, datasetA, parametersA, start, stop, **opts)\r\ndataB, metaB = hapi(serverB, datasetB, parametersB, start, stop, **opts)\r\ndataB = dataB[80:100]\r\nnewAB, metaAB = hapitools.merge_hapi(dataA, metaA, dataB, metaB, round_to_sec = True, fill_nan=False)\r\nhapiplot(newAB, metaAB)\r\n```\r\n```\r\n---------------------------------------------------------------------------\r\nValueError                                Traceback (most recent call last)\r\nCell In[2], [line 14](vscode-notebook-cell:?execution_count=2\u0026line=14)\r\n     [12](vscode-notebook-cell:?execution_count=2\u0026line=12) dataB, metaB = hapi(serverB, datasetB, parametersB, start, stop, **opts)\r\n     [13](vscode-notebook-cell:?execution_count=2\u0026line=13) dataB = dataB[80:100]\r\n---\u003e [14](vscode-notebook-cell:?execution_count=2\u0026line=14) newAB, metaAB = hapitools.merge_hapi(dataA, metaA, dataB, metaB, round_to_sec = True, fill_nan=False)\r\n     [15](vscode-notebook-cell:?execution_count=2\u0026line=15) hapiplot(newAB, metaAB)\r\n\r\nFile ~/GitHub/hapi-server/tools-python/src/hapitools.py:123, in merge_hapi(data1, meta1, data2, meta2, how, round_to_sec, fill_nan)\r\n    [121](https://file+.vscode-resource.vscode-cdn.net/Users/tinsmcl1/GitHub/hapi-server/tools-python/~/GitHub/hapi-server/tools-python/src/hapitools.py:121) dt = merge_dtypes(data1, data2, trim='Time')\r\n    [122](https://file+.vscode-resource.vscode-cdn.net/Users/tinsmcl1/GitHub/hapi-server/tools-python/~/GitHub/hapi-server/tools-python/src/hapitools.py:122) new_data = new_df.to_records(index=False, column_dtypes={\"Time\": \"S30\"})\r\n--\u003e [123](https://file+.vscode-resource.vscode-cdn.net/Users/tinsmcl1/GitHub/hapi-server/tools-python/~/GitHub/hapi-server/tools-python/src/hapitools.py:123) new_data = np.array([tuple([nparray_unpack_to_list(e) for e in elm]) for elm in new_data], dtype=dt)\r\n    [124](https://file+.vscode-resource.vscode-cdn.net/Users/tinsmcl1/GitHub/hapi-server/tools-python/~/GitHub/hapi-server/tools-python/src/hapitools.py:124) new_data = np.array([tuple(i) for i in new_data], dtype=dt)\r\n    [126](https://file+.vscode-resource.vscode-cdn.net/Users/tinsmcl1/GitHub/hapi-server/tools-python/~/GitHub/hapi-server/tools-python/src/hapitools.py:126) return new_data, new_meta\r\n\r\nValueError: cannot convert float NaN to integer\r\n```\r\n","author":{"url":"https://github.com/tinsmcl1","@type":"Person","name":"tinsmcl1"},"datePublished":"2024-07-31T15:46:31.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":4},"url":"https://github.com/4/tools-python/issues/4"}