René's URL Explorer Experiment

Title: bpo-44689: ctypes.util.find_library() now finds macOS 11+ system libraries when built on older macOS systems by bergkvist · Pull Request #27251 · python/cpython · GitHub

Open Graph Title: bpo-44689: ctypes.util.find_library() now finds macOS 11+ system libraries when built on older macOS systems by bergkvist · Pull Request #27251 · python/cpython

X Title: bpo-44689: ctypes.util.find_library() now finds macOS 11+ system libraries when built on older macOS systems by bergkvist · Pull Request #27251 · python/cpython

Description: Improve binary portability between MacOS versions https://bugs.python.org/issue44689 When compiling CPython using a MacOS Catalina build server, it won't work as expected when trying to run it on MacOS Big Sur. In particular, ctypes.util.find_library(name) will always return None. This PR adds support for compiling on an older MacOS version. Background Checking if a shared library exists on MacOS Big Sur is no longer possible by looking at the file system. Instead, Apple recommends the use of dlopen to check if a shared library exists (MacOS Big Sur 11.0.1 changenotes). Note that using dlopen to check for library existence is not ideal, since this might cause arbitrary side effects. _dyld_shared_cache_contains_path and build time requirements The current solution (introduced in #22855) to make find_library work on MacOS Big Sur does not use dlopen, but rather _dyld_shared_cache_contains_path from #include . This function/symbol is only available on MacOS Big Sur, meaning it will only be included if you compile CPython on MacOS Big Sur. In other words, if we compile CPython on MacOS Catalina, and move the binary to MacOS Big Sur, from _ctypes import _dyld_shared_cache_contains_path will raise an ImportError - and find_library will not work, always returning None. # ctypes/macholib/dyld.py # ... +try: + from _ctypes import _dyld_shared_cache_contains_path +except ImportError: + def _dyld_shared_cache_contains_path(*args): + raise NotImplementedError # ... def dyld_find(name, executable_path=None, env=None): """ Find a library or framework using dyld semantics """ for path in dyld_image_suffix_search(chain( dyld_override_search(name, env), dyld_executable_path_search(name, executable_path), dyld_default_search(name, env), ), env): if os.path.isfile(path): return path + try: + if _dyld_shared_cache_contains_path(path): + return path + except NotImplementedError: + pass # ... Method/Verification In order to support building for MacOS >= 11.0 (Big Sur or later) with a build server running MacOS < 11.0 (Catalina or earlier), we use dynamic loading to avoid errors when compiling. _dyld_shared_cache_contains_path will be resolved at runtime instead of compile-time, since it is not available at compile-time in this case. The downside to doing this at runtime instead of compile-time is that the compiler won't give us useful error messages. This is why we want to keep using the weak linking approach when compiling on MacOS >= 11.0 (Big Sur or later) - and only use dynamic loading to support the deprecated use case of compiling on an older MacOS version. To test that this works, I've set up 2 virtual machines using OSX-KVM on a Manjaro Linux host: Catalina (OS: macOS Catalina 10.15.7 19H15 x86_64, Kernel 19.6.0) Big Sur (OS: macOS 11.4 20F71 x86_64, Kernel: 20.5.0) Files are copied between the 2 virtual machines using rsync: #!/usr/bin/env bash rm -rf "./local" echo "Pulling from Catalina... (/usr/local/ -> ./local)" rsync -a -e "ssh -p 2222" tobias@localhost:/usr/local . echo "Pushing to BigSur... (./local/ -> /usr/local)" rsync -a -e "ssh -p 3222" ./local/* tobias@localhost:/usr/local/ echo "Done!" Checks Compile on Catalina and install, copy /usr/local/* over to BigSur, and call find_library('c') Results Before this change: ❌ Compile on Catalina and install, copy /usr/local/* over to BigSur, and call find_library('c') Python 3.11.0a0 (heads/main:635bfe8162, Jul 19 2021, 08:09:05) [Clang 12.0.0 (clang-1200.0.32.29)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> from ctypes.util import find_library; print(find_library('c')) None After this change: ✔️ Compile on Catalina and install, copy /usr/local/* over to BigSur, and call find_library('c') Python 3.11.0a0 (heads/macos-ctypes-find-library:0858c0c9ff, Jul 19 2021, 17:52:24) [Clang 12.0.0 (clang-1200.0.32.29)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> from ctypes.util import find_library; print(find_library('c')) /usr/lib/libc.dylib Related issues/Symptoms of the issue Note that some of these are closed due to the symptom having been treated - rather than the underlying cause. NixOS/nixpkgs#105038 jupyterlab/jupyterlab#9863 arsenetar/send2trash#51 minrk/appnope#12 jupyterlab/jupyterlab#9410 hbldh/bleak#372 https://bugs.python.org/issue43052 https://stackoverflow.com/questions/65065482/error-when-importing-pyautogui-on-macos-10-13-and-python-3-8 https://bugs.python.org/issue44689

Open Graph Description: Improve binary portability between MacOS versions https://bugs.python.org/issue44689 When compiling CPython using a MacOS Catalina build server, it won't work as expected when trying to run it ...

X Description: Improve binary portability between MacOS versions https://bugs.python.org/issue44689 When compiling CPython using a MacOS Catalina build server, it won&#39;t work as expected when trying to run...

Opengraph URL: https://github.com/python/cpython/pull/27251

X: @github

direct link

Domain: github.com

Links:

Viewport: width=device-width