René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"pythonnet.load crashes if the Python interpreter is called from PATH in Ubuntu 20.04","articleBody":"### Environment\r\n\r\n-   Pythonnet version: 3.0.0-dev (latest master on 18/2/2021)\r\n-   Python version: 3.8.5\r\n-   Operating System: Ubuntu 20.04\r\n-   .NET Runtime: .NET 5.0.2\r\n\r\n### Details\r\n\r\n-   Describe what you were trying to get done.\r\nI am trying to use the latest `pythonnet` in a virtual environment. If I call my simple demo script using `python` command in PATH, pythonnet.load() crashes. It is the same with any other PATH-based `pythonX` command. Using a full path works fine. See the details and the analysis below.\r\n\r\nPreparation\r\n```\r\n# Create virtual env.\r\npython3 -m virtualenv venv\r\n. venv/bin/activate\r\n# Install the latest pythonnet\r\ngit clone https://github.com/pythonnet/pythonnet.git\r\npip install ./pythonnet/\r\n# Get .NET 5 runtime\r\nwget https://download.visualstudio.microsoft.com/download/pr/824175f2-5577-46ec-a675-95d2f652575f/07dafc2398e669afa7d25f2d47398c36/dotnet-runtime-5.0.2-linux-x64.tar.gz\r\nmkdir dotnet_5.0.2\r\ntar xfzv dotnet-runtime-5.0.2-linux-x64.tar.gz -C dotnet_5.0.2/\r\n```\r\n\r\nRuntime config (`cat runtimeconfig.json`)\r\n``` \r\n{\r\n  \"runtimeOptions\": {\r\n    \"tfm\": \"net5.0\",\r\n    \"framework\": {\r\n      \"name\": \"Microsoft.NETCore.App\",\r\n      \"version\": \"5.0.2\"\r\n    }\r\n  }\r\n}\r\n```\r\n\r\nPython script (`cat bug.py`):\r\n```python\r\nimport os\r\n\r\nimport clr_loader\r\nimport pythonnet\r\n\r\nruntime_config = os.path.join(os.path.dirname(__file__), \"runtimeconfig.json\")\r\ndotnet_root = os.path.join(os.path.dirname(__file__), \"dotnet_5.0.2\")\r\n\r\nruntime = clr_loader.get_coreclr(runtime_config=runtime_config, dotnet_root=dotnet_root)\r\n\r\npythonnet.set_runtime(runtime)\r\npythonnet.load()\r\n\r\nimport System\r\nfrom System import String\r\n\r\ns = String(\"Hello World from Python.NET for .NET 5!\")\r\nSystem.Console.WriteLine(s)\r\n```\r\n\r\n-   What commands did you run to trigger this issue? If you can provide a\r\n    [Minimal, Complete, and Verifiable example](http://stackoverflow.com/help/mcve)\r\n    this will help us understand the issue.\r\n\r\n```\r\n\u003e which python\r\n/home/takacs/src/pythonnet/venv/bin/python\r\n\r\n\u003e`which python` bug.py \r\nHello World from Python.NET for .NET 5!\r\n \r\n\u003e python bug.py \r\nTraceback (most recent call last):\r\n  File \"bug.py\", line 12, in \u003cmodule\u003e\r\n    pythonnet.load()\r\n  File \"/home/takacs/src/pythonnet/venv/lib/python3.8/site-packages/pythonnet/__init__.py\", line 50, in load\r\n    _FFI.dlopen(libpython, posix.RTLD_NODELETE | posix.RTLD_LOCAL)\r\n  File \"/home/takacs/src/pythonnet/venv/lib/python3.8/site-packages/cffi/api.py\", line 150, in dlopen\r\n    lib, function_cache = _make_ffi_library(self, name, flags)\r\n  File \"/home/takacs/src/pythonnet/venv/lib/python3.8/site-packages/cffi/api.py\", line 832, in _make_ffi_library\r\n    backendlib = _load_backend_lib(backend, libname, flags)\r\n  File \"/home/takacs/src/pythonnet/venv/lib/python3.8/site-packages/cffi/api.py\", line 827, in _load_backend_lib\r\n    raise OSError(msg)\r\nOSError: cannot load library '/home/takacs/src/pythonnet/python': /home/takacs/src/pythonnet/python: cannot open shared object file: No such file or directory.  Additionally, ctypes.util.find_library() did not manage to locate a library called '/home/takacs/src/pythonnet/python'\r\n```\r\n\r\n-   If there was a crash, please include the traceback here.\r\n```\r\nTraceback (most recent call last):\r\n  File \"bug.py\", line 12, in \u003cmodule\u003e\r\n    pythonnet.load()\r\n  File \"/home/takacs/src/pythonnet/venv/lib/python3.8/site-packages/pythonnet/__init__.py\", line 50, in load\r\n    _FFI.dlopen(libpython, posix.RTLD_NODELETE | posix.RTLD_LOCAL)\r\n  File \"/home/takacs/src/pythonnet/venv/lib/python3.8/site-packages/cffi/api.py\", line 150, in dlopen\r\n    lib, function_cache = _make_ffi_library(self, name, flags)\r\n  File \"/home/takacs/src/pythonnet/venv/lib/python3.8/site-packages/cffi/api.py\", line 832, in _make_ffi_library\r\n    backendlib = _load_backend_lib(backend, libname, flags)\r\n  File \"/home/takacs/src/pythonnet/venv/lib/python3.8/site-packages/cffi/api.py\", line 827, in _load_backend_lib\r\n    raise OSError(msg)\r\nOSError: cannot load library '/home/takacs/src/pythonnet/python': /home/takacs/src/pythonnet/python: cannot open shared object file: No such file or directory.  Additionally, ctypes.util.find_library() did not manage to locate a library called \r\n```\r\n\r\n### Analysis\r\n\r\nThis issue is in the `pythonnet.find_libpython` module. The `dlinfo.dli_fname` string contains the actual command that was used to start the Python interpreter. In the case of pure `python` command, `os.path.realpath` fails the properly translate this back to the real executable.\r\n\r\nhttps://github.com/pythonnet/pythonnet/blob/0f5e7814c68b846cd89019f2e23fed69eaa59eca/pythonnet/find_libpython/__init__.py#L90)\r\n\r\nAdding a few debug lines properly demonstrate this:\r\n```Python\r\ndef _linked_libpython_unix():\r\n    libdl = ctypes.CDLL(ctypes.util.find_library(\"dl\"))\r\n    libdl.dladdr.argtypes = [ctypes.c_void_p, ctypes.POINTER(Dl_info)]\r\n    libdl.dladdr.restype = ctypes.c_int\r\n\r\n    dlinfo = Dl_info()\r\n    retcode = libdl.dladdr(\r\n        ctypes.cast(ctypes.pythonapi.Py_GetVersion, ctypes.c_void_p),\r\n        ctypes.pointer(dlinfo))\r\n    if retcode == 0:  # means error\r\n        return None\r\n\r\n    print(f\"DEBUG dlinfo.dli_fname.decode(): {dlinfo.dli_fname.decode()}\")\r\n    path = os.path.realpath(dlinfo.dli_fname.decode())\r\n    print(f\"DEBUG path: {path}\")\r\n    \r\n    if path == os.path.realpath(sys.executable):\r\n        return None\r\n    return path\r\n```\r\n```\r\n\u003e  ./venv/bin/python bug.py \r\nDEBUG dlinfo.dli_fname.decode(): ./venv/bin/python\r\nDEBUG path: /usr/bin/python3.8\r\nHello World from Python.NET for .NET 5!\r\n\r\n\u003e ../pythonnet/venv/bin/python bug.py \r\nDEBUG dlinfo.dli_fname.decode(): ../pythonnet/venv/bin/python\r\nDEBUG path: /usr/bin/python3.8\r\nHello World from Python.NET for .NET 5!\r\n\r\n\u003epython bug.py \r\nDEBUG dlinfo.dli_fname.decode(): python\r\nDEBUG path: /home/takacs/src/pythonnet/python\r\nTraceback (most recent call last):\r\n...\r\n```\r\nThe last case demonstrates that `os.path.realpath` just extends `python` with the current path which is obviously incorrect.\r\n\r\nAs a result, `pythonnet.load()` method's `libpython` variable will point to a library which cannot be loaded (https://github.com/pythonnet/pythonnet/blob/0f5e7814c68b846cd89019f2e23fed69eaa59eca/pythonnet/__init__.py#L50).\r\n However, if this part is disabled, the application will crash later in the .NET `Python.Runtime` for the same reason.\r\n","author":{"url":"https://github.com/tibortakacs","@type":"Person","name":"tibortakacs"},"datePublished":"2021-02-18T19:44:32.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":16},"url":"https://github.com/1388/pythonnet/issues/1388"}