René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Error compiling HACL* Blake2 support for macOS universal binaries","articleBody":"# Bug report\n\n### Bug description:\n\n#99108 tracks the addition of a native HACL implementation to CPython. #119316 added an implementation of Blake2 to `hashlib`.\n\nThis compiles fine on single architecture macOS builds (as verified by CI); but universal2 builds running on an ARM64 laptop generate a compilation error:\n\nTo reproduce the problem: on a macOS machine, configure the build with:\n```\n$ configure --enable-universalsdk=\"`xcrun --show-sdk-path`\" --with-universal-archs=universal2\n$ make\n```\n\nThis will eventually yield the compilation error:\n```\ngcc -c -I../../../Modules/_hacl -I../../../Modules/_hacl/include -D_BSD_SOURCE -D_DEFAULT_SOURCE -fno-strict-overflow -Wsign-compare -Wunreachable-code -DNDEBUG -g -O3 -Wall -arch arm64 -arch x86_64 -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk    -fstack-protector-strong -std=c11 -Wextra -Wno-unused-parameter -Wno-missing-field-initializers -Wstrict-prototypes -Werror=implicit-function-declaration -fvisibility=hidden  -I../../../Include/internal -I../../../Include/internal/mimalloc -IObjects -IInclude -IPython -I. -I../../../Include -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk     -mavx2 -DHACL_CAN_COMPILE_VEC256 -o Modules/_hacl/Hacl_Hash_Blake2b_Simd256.o ../../../Modules/_hacl/Hacl_Hash_Blake2b_Simd256.c\nIn file included from ../../../Modules/_hacl/Hacl_Hash_Blake2b_Simd256.c:26:\nIn file included from ../../../Modules/_hacl/internal/Hacl_Hash_Blake2b_Simd256.h:40:\n../../../Modules/_hacl/internal/../Hacl_Hash_Blake2b_Simd256.h:56:3: error: unknown type name 'Lib_IntVector_Intrinsics_vec256'\n  Lib_IntVector_Intrinsics_vec256 *fst;\n  ^\n../../../Modules/_hacl/internal/../Hacl_Hash_Blake2b_Simd256.h:57:3: error: unknown type name 'Lib_IntVector_Intrinsics_vec256'\n  Lib_IntVector_Intrinsics_vec256 *snd;\n  ^\nIn file included from ../../../Modules/_hacl/Hacl_Hash_Blake2b_Simd256.c:26:\n../../../Modules/_hacl/internal/Hacl_Hash_Blake2b_Simd256.h:44:32: error: unknown type name 'Lib_IntVector_Intrinsics_vec256'\nHacl_Hash_Blake2b_Simd256_init(Lib_IntVector_Intrinsics_vec256 *hash, uint32_t kk, uint32_t nn);\n                               ^\n../../../Modules/_hacl/internal/Hacl_Hash_Blake2b_Simd256.h:49:3: error: unknown type name 'Lib_IntVector_Intrinsics_vec256'\n  Lib_IntVector_Intrinsics_vec256 *wv,\n  ^\n../../../Modules/_hacl/internal/Hacl_Hash_Blake2b_Simd256.h:50:3: error: unknown type name 'Lib_IntVector_Intrinsics_vec256'\n  Lib_IntVector_Intrinsics_vec256 *hash,\n  ^\n../../../Modules/_hacl/internal/Hacl_Hash_Blake2b_Simd256.h:59:3: error: unknown type name 'Lib_IntVector_Intrinsics_vec256'\n  Lib_IntVector_Intrinsics_vec256 *wv,\n  ^\n../../../Modules/_hacl/internal/Hacl_Hash_Blake2b_Simd256.h:60:3: error: unknown type name 'Lib_IntVector_Intrinsics_vec256'\n  Lib_IntVector_Intrinsics_vec256 *hash,\n  ^\n../../../Modules/_hacl/internal/Hacl_Hash_Blake2b_Simd256.h:71:3: error: unknown type name 'Lib_IntVector_Intrinsics_vec256'\n  Lib_IntVector_Intrinsics_vec256 *hash\n  ^\n../../../Modules/_hacl/internal/Hacl_Hash_Blake2b_Simd256.h:76:3: error: unknown type name 'Lib_IntVector_Intrinsics_vec256'\n  Lib_IntVector_Intrinsics_vec256 *st,\n  ^\n../../../Modules/_hacl/internal/Hacl_Hash_Blake2b_Simd256.h:83:3: error: unknown type name 'Lib_IntVector_Intrinsics_vec256'\n  Lib_IntVector_Intrinsics_vec256 *st\n  ^\n../../../Modules/_hacl/internal/Hacl_Hash_Blake2b_Simd256.h:86:1: error: unknown type name 'Lib_IntVector_Intrinsics_vec256'\nLib_IntVector_Intrinsics_vec256 *Hacl_Hash_Blake2b_Simd256_malloc_with_key(void);\n^\n../../../Modules/_hacl/Hacl_Hash_Blake2b_Simd256.c:34:3: error: unknown type name 'Lib_IntVector_Intrinsics_vec256'\n  Lib_IntVector_Intrinsics_vec256 *wv,\n  ^\n../../../Modules/_hacl/Hacl_Hash_Blake2b_Simd256.c:35:3: error: unknown type name 'Lib_IntVector_Intrinsics_vec256'\n  Lib_IntVector_Intrinsics_vec256 *hash,\n  ^\n../../../Modules/_hacl/Hacl_Hash_Blake2b_Simd256.c:53:3: error: use of undeclared identifier 'Lib_IntVector_Intrinsics_vec256'\n  Lib_IntVector_Intrinsics_vec256 mask = Lib_IntVector_Intrinsics_vec256_zero;\n  ^\n../../../Modules/_hacl/Hacl_Hash_Blake2b_Simd256.c:72:3: error: use of undeclared identifier 'mask'\n  mask =\n  ^\n../../../Modules/_hacl/Hacl_Hash_Blake2b_Simd256.c:73:5: error: call to undeclared function 'Lib_IntVector_Intrinsics_vec256_load64s'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]\n    Lib_IntVector_Intrinsics_vec256_load64s(FStar_UInt128_uint128_to_uint64(totlen),\n    ^\n../../../Modules/_hacl/Hacl_Hash_Blake2b_Simd256.c:77:33: error: use of undeclared identifier 'Lib_IntVector_Intrinsics_vec256'; did you mean 'Lib_IntVector_Intrinsics_vec256_load64s'?\n  memcpy(wv, hash, 4U * sizeof (Lib_IntVector_Intrinsics_vec256));\n                                ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n                                Lib_IntVector_Intrinsics_vec256_load64s\n/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/secure/_string.h:63:33: note: expanded from macro 'memcpy'\n                __builtin___memcpy_chk (dest, __VA_ARGS__, __darwin_obsz0 (dest))\n                                              ^\n../../../Modules/_hacl/Hacl_Hash_Blake2b_Simd256.c:73:5: note: 'Lib_IntVector_Intrinsics_vec256_load64s' declared here\n    Lib_IntVector_Intrinsics_vec256_load64s(FStar_UInt128_uint128_to_uint64(totlen),\n    ^\n../../../Modules/_hacl/Hacl_Hash_Blake2b_Simd256.c:78:3: error: use of undeclared identifier 'Lib_IntVector_Intrinsics_vec256'; did you mean 'Lib_IntVector_Intrinsics_vec256_load64s'?\n  Lib_IntVector_Intrinsics_vec256 *wv3 = wv + 3U;\n  ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n  Lib_IntVector_Intrinsics_vec256_load64s\n../../../Modules/_hacl/Hacl_Hash_Blake2b_Simd256.c:73:5: note: 'Lib_IntVector_Intrinsics_vec256_load64s' declared here\n    Lib_IntVector_Intrinsics_vec256_load64s(FStar_UInt128_uint128_to_uint64(totlen),\n    ^\n../../../Modules/_hacl/Hacl_Hash_Blake2b_Simd256.c:78:36: error: use of undeclared identifier 'wv3'\n  Lib_IntVector_Intrinsics_vec256 *wv3 = wv + 3U;\n                                   ^\nfatal error: too many errors emitted, stopping now [-ferror-limit=]\n20 errors generated.\nmake: *** [Modules/_hacl/Hacl_Hash_Blake2b_Simd256.o] Error 1\n```\n\nFrom what I can make out, the error comes from the detection of `-mavx2` support. On a bare configure on an ARM64 machine, `-mavx2` support is apparently unsupported:\n```\nconfigure:30537: checking whether C compiler accepts -mavx2\nconfigure:30557: gcc -c  -Werror -mavx2  conftest.c \u003e\u00265\nclang: error: argument unused during compilation: '-mavx2' [-Werror,-Wunused-command-line-argument]\n```\nand as a result, the `Hacl_Hash_Blake2b_Simd256.c` module isn't compiled. However, when universal support is enabled, `-mavx2` *is* supported:\n```\nconfigure:30537: checking whether C compiler accepts -mavx2\nconfigure:30557: gcc -c -arch arm64 -arch x86_64 -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk  -Werror -mavx2 -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk  conftest.c \u003e\u00265\nconfigure:30557: $? = 0\nconfigure:30566: result: yes\n```\nand the module is included. Based on recent configure logs for x86_64 macOS builds, it appears that `-mavx2` *is* supported on x86_64. \n\nI'm not sufficiently familiar with the subject matter to comment on whether the fix here is to fix the autoconf detection to *disable* the problematic module on universal builds, or to correct the implementation so that it can compile for universal builds.\n\nTagging @msprotz @R1kM as the authors of the recent HACL* changes.\n\n### CPython versions tested on:\n\nCPython main branch\n\n### Operating systems tested on:\n\nmacOS\n\n\u003c!-- gh-linked-prs --\u003e\n### Linked PRs\n* gh-123927\n* gh-123989\n\u003c!-- /gh-linked-prs --\u003e\n","author":{"url":"https://github.com/freakboy3742","@type":"Person","name":"freakboy3742"},"datePublished":"2024-09-05T22:56:03.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":22},"url":"https://github.com/123748/cpython/issues/123748"}