René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"segfaults when using go feature server","articleBody":"## Expected Behavior \r\n\r\nThe go feature server should not segfault\r\n\r\n## Current Behavior\r\n\r\nWith the following diff:\r\n```\r\ndiff --git a/sdk/python/tests/integration/online_store/test_universal_online.py b/sdk/python/tests/integration/online_store/test_universal_online.py\r\nindex b01448e7..212611b8 100644\r\n--- a/sdk/python/tests/integration/online_store/test_universal_online.py\r\n+++ b/sdk/python/tests/integration/online_store/test_universal_online.py\r\n@@ -524,6 +524,9 @@ def test_stream_feature_view_online_retrieval(\r\n def test_online_retrieval(\r\n     environment, universal_data_sources, feature_server_endpoint, full_feature_names\r\n ):\r\n+\r\n+    import faulthandler; faulthandler.enable()\r\n+\r\n     fs = environment.feature_store\r\n     entities, datasets, data_sources = universal_data_sources\r\n     feature_views = construct_universal_feature_views(data_sources)\r\n@@ -620,13 +623,17 @@ def test_online_retrieval(\r\n     unprefixed_feature_refs.remove(\"conv_rate_plus_100\")\r\n     unprefixed_feature_refs.remove(\"conv_rate_plus_val_to_add\")\r\n\r\n-    online_features_dict = get_online_features_dict(\r\n-        environment=environment,\r\n-        endpoint=feature_server_endpoint,\r\n-        features=feature_refs,\r\n-        entity_rows=entity_rows,\r\n-        full_feature_names=full_feature_names,\r\n-    )\r\n+    import psutil\r\n+\r\n+    for i in range(1000):\r\n+        print(psutil.virtual_memory())\r\n+        online_features_dict = get_online_features_dict(\r\n+            environment=environment,\r\n+            endpoint=feature_server_endpoint,\r\n+            features=feature_refs,\r\n+            entity_rows=entity_rows,\r\n+            full_feature_names=full_feature_names,\r\n+        )\r\n```\r\n\r\nIf I run `test_online_retrieval`, I can reliable segfault and terminate the pytest process.\r\n\r\nWith some digging, I was able to get a stack trace that looks like this:\r\n\r\n```\r\nruntime: pointer 0xc0006b0151 to unallocated span span.base()=0xc0006b0000 span.limit=0xc0006b2000 span.state=0\r\nfatal error: found bad pointer in Go heap (incorrect use of unsafe or cgo?)\r\n\r\nruntime stack:\r\nruntime.throw({0x144b9685e?, 0x10?})\r\n\t/usr/local/Cellar/go/1.18.3/libexec/src/runtime/panic.go:992 +0x71 fp=0x7ff7b3210460 sp=0x7ff7b3210430 pc=0x143d19df1\r\nruntime.badPointer(0x142bfb068, 0xc0004f75c0?, 0x0, 0x7ff7b32104f0?)\r\n\t/usr/local/Cellar/go/1.18.3/libexec/src/runtime/mbitmap.go:368 +0xa5 fp=0x7ff7b32104b0 sp=0x7ff7b3210460 pc=0x143cf7a25\r\nruntime.findObject(0xc0006b0151, 0xd0?, 0x7ff7b3210530?)\r\n\t/usr/local/Cellar/go/1.18.3/libexec/src/runtime/mbitmap.go:410 +0xed fp=0x7ff7b3210500 sp=0x7ff7b32104b0 pc=0x143cf7cad\r\nruntime.wbBufFlush1(0xc00004c000)\r\n\t/usr/local/Cellar/go/1.18.3/libexec/src/runtime/mwbbuf.go:260 +0xe5 fp=0x7ff7b3210570 sp=0x7ff7b3210500 pc=0x143d14f25\r\nruntime.wbBufFlush.func1()\r\n\t/usr/local/Cellar/go/1.18.3/libexec/src/runtime/mwbbuf.go:201 +0x2a fp=0x7ff7b3210588 sp=0x7ff7b3210570 pc=0x143d14d6a\r\nruntime.systemstack()\r\n\t/usr/local/Cellar/go/1.18.3/libexec/src/runtime/asm_amd64.s:469 +0x49 fp=0x7ff7b3210590 sp=0x7ff7b3210588 pc=0x143d49329\r\n\r\ngoroutine 17 [running, locked to thread]:\r\nruntime.systemstack_switch()\r\n\t/usr/local/Cellar/go/1.18.3/libexec/src/runtime/asm_amd64.s:436 fp=0xc00071ad58 sp=0xc00071ad50 pc=0x143d492c0\r\nruntime.wbBufFlush(0x1428ec923?, 0x143cef65e?)\r\n\t/usr/local/Cellar/go/1.18.3/libexec/src/runtime/mwbbuf.go:200 +0x6e fp=0xc00071ad78 sp=0xc00071ad58 pc=0x143d14dee\r\nruntime.bulkBarrierPreWrite(0xc00071ae20?, 0xc00071af40, 0x10)\r\n\t/usr/local/Cellar/go/1.18.3/libexec/src/runtime/mbitmap.go:620 +0x271 fp=0xc00071adf8 sp=0xc00071ad78 pc=0x143cf8371\r\nruntime.typedmemmove(0x144ccb0e0, 0xc000992460, 0xc00071af40)\r\n\t/usr/local/Cellar/go/1.18.3/libexec/src/runtime/mbarrier.go:162 +0x45 fp=0xc00071ae30 sp=0xc00071adf8 pc=0x143cf7125\r\nsync.(*Map).Store(0x1453da440, {0x144ca1480, 0xc00062ce20}, {0x144d763a0, 0xc00059e840})\r\n\t/usr/local/Cellar/go/1.18.3/libexec/src/sync/map.go:137 +0x91 fp=0xc00071af28 sp=0xc00071ae30 pc=0x143d4fc51\r\ngithub.com/apache/arrow/go/v8/arrow/cdata.storeData({0x144e032d8, 0xc00059e840})\r\n\t/Users/achal/go/pkg/mod/github.com/apache/arrow/go/v8@v8.0.0-20220408212425-58fe60f59289/arrow/cdata/exports.go:45 +0xc5 fp=0xc00071af90 sp=0xc00071af28 pc=0x1441674e5\r\ngithub.com/apache/arrow/go/v8/arrow/cdata.exportArray({0x144e044c0, 0xc0005ad6c0}, 0x7fd7443c0990, 0x0)\r\n\t/Users/achal/go/pkg/mod/github.com/apache/arrow/go/v8@v8.0.0-20220408212425-58fe60f59289/arrow/cdata/cdata_exports.go:356 +0x365 fp=0xc00071b260 sp=0xc00071af90 pc=0x144166945\r\ngithub.com/apache/arrow/go/v8/arrow/cdata.exportArray({0x144e04800, 0xc0005ad540}, 0x6000025299b0, 0x0)\r\n\t/Users/achal/go/pkg/mod/github.com/apache/arrow/go/v8@v8.0.0-20220408212425-58fe60f59289/arrow/cdata/cdata_exports.go:385 +0xae7 fp=0xc00071b530 sp=0xc00071b260 pc=0x1441670c7\r\ngithub.com/apache/arrow/go/v8/arrow/cdata.ExportArrowRecordBatch({0x144e04b40, 0xc00081fc20}, 0x6000025299b0, 0x6000025298b0)\r\n\t/Users/achal/go/pkg/mod/github.com/apache/arrow/go/v8@v8.0.0-20220408212425-58fe60f59289/arrow/cdata/interface.go:217 +0x413 fp=0xc00071b740 sp=0xc00071b530 pc=0x14415c433\r\ngithub.com/feast-dev/feast/go/embedded.(*OnlineFeatureService).GetOnlineFeatures(0xc0004b0000, {0xc0000b8c00, 0xa, 0x10}, {0x0, 0x0}, {0x600002529a30, 0x600002528530}, {0x600002529db0, 0x60000252a430}, ...)\r\n\t/Users/achal/tecton/feast/go/embedded/online_features.go:220 +0xef0 fp=0xc00071bbc8 sp=0xc00071b740 pc=0x1449fa470\r\nmain.embedded_OnlineFeatureService_GetOnlineFeatures(0x2, 0x3fd, 0x10d233320, 0x3fe, 0x3ff, 0x1, 0x400)\r\n\t/Users/achal/tecton/feast/build/lib.macosx-12.0-x86_64-3.8/feast/embedded_go/lib/embedded.go:1350 +0x365 fp=0xc00071bda0 sp=0xc00071bbc8 pc=0x144a063e5\r\n_cgoexp_79ecca462223_embedded_OnlineFeatureService_GetOnlineFeatures(0x7ff7b3210608)\r\n\t_cgo_gotypes.go:1936 +0xcd fp=0xc00071be38 sp=0xc00071bda0 pc=0x144a0ec6d\r\nruntime.cgocallbackg1(0x144a0eba0, 0xc00008ffe0?, 0x0)\r\n\t/usr/local/Cellar/go/1.18.3/libexec/src/runtime/cgocall.go:314 +0x2b3 fp=0xc00071bf00 sp=0xc00071be38 pc=0x143ce5673\r\nruntime.cgocallbackg(0x0?, 0x0?, 0x0?)\r\n\t/usr/local/Cellar/go/1.18.3/libexec/src/runtime/cgocall.go:233 +0xf9 fp=0xc00071bf90 sp=0xc00071bf00 pc=0x143ce5319\r\nruntime.cgocallbackg(0x144a0eba0, 0x7ff7b3210608, 0x0)\r\n\t\u003cautogenerated\u003e:1 +0x2f fp=0xc00071bfb8 sp=0xc00071bf90 pc=0x143d4d78f\r\nruntime.cgocallback(0x0, 0x0, 0x0)\r\n\t/usr/local/Cellar/go/1.18.3/libexec/src/runtime/asm_amd64.s:971 +0xb4 fp=0xc00071bfe0 sp=0xc00071bfb8 pc=0x143d4b2f4\r\nruntime.goexit()\r\n\t/usr/local/Cellar/go/1.18.3/libexec/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc00071bfe8 sp=0xc00071bfe0 pc=0x143d4b501\r\n\r\ngoroutine 35 [select]:\r\nruntime.gopark(0x144df51f8, 0x0, 0x9, 0x18, 0x1)\r\n\t/usr/local/Cellar/go/1.18.3/libexec/src/runtime/proc.go:361 +0xf2 fp=0xc000123cd8 sp=0xc000123ca8 pc=0x143d1c892\r\nruntime.selectgo(0xc000123f80, 0xc000123e80, 0x0?, 0x0, 0x0?, 0x1)\r\n\t/usr/local/Cellar/go/1.18.3/libexec/src/runtime/select.go:328 +0xa45 fp=0xc000123e30 sp=0xc000123cd8 pc=0x143d2bf65\r\ngithub.com/go-redis/redis/v8/internal/pool.(*ConnPool).reaper(0xc000355400, 0xdf8475800)\r\n\t/Users/achal/go/pkg/mod/github.com/go-redis/redis/v8@v8.11.4/internal/pool/pool.go:485 +0x15d fp=0xc000123fb0 sp=0xc000123e30 pc=0x14462025d\r\ngithub.com/go-redis/redis/v8/internal/pool.NewConnPool.func1()\r\n\t/Users/achal/go/pkg/mod/github.com/go-redis/redis/v8@v8.11.4/internal/pool/pool.go:111 +0x39 fp=0xc000123fe0 sp=0xc000123fb0 pc=0x14461d199\r\nruntime.goexit()\r\n\t/usr/local/Cellar/go/1.18.3/libexec/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc000123fe8 sp=0xc000123fe0 pc=0x143d4b501\r\ncreated by github.com/go-redis/redis/v8/internal/pool.NewConnPool\r\n\t/Users/achal/go/pkg/mod/github.com/go-redis/redis/v8@v8.11.4/internal/pool/pool.go:111 +0x36e\r\n\r\ngoroutine 2 [force gc (idle)]:\r\nruntime.gopark(0x144df51a0, 0x1453d9ff0, 0x11, 0x14, 0x1)\r\n\t/usr/local/Cellar/go/1.18.3/libexec/src/runtime/proc.go:361 +0xf2 fp=0xc00007af88 sp=0xc00007af58 pc=0x143d1c892\r\nruntime.goparkunlock(0x0?, 0x0?, 0x0?, 0x0?)\r\n\t/usr/local/Cellar/go/1.18.3/libexec/src/runtime/proc.go:367 +0x2a fp=0xc00007afb8 sp=0xc00007af88 pc=0x143d1c92a\r\nruntime.forcegchelper()\r\n\t/usr/local/Cellar/go/1.18.3/libexec/src/runtime/proc.go:301 +0xa5 fp=0xc00007afe0 sp=0xc00007afb8 pc=0x143d1c6c5\r\nruntime.goexit()\r\n\t/usr/local/Cellar/go/1.18.3/libexec/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc00007afe8 sp=0xc00007afe0 pc=0x143d4b501\r\ncreated by runtime.init.6\r\n\t/usr/local/Cellar/go/1.18.3/libexec/src/runtime/proc.go:289 +0x25\r\n\r\ngoroutine 18 [GC sweep wait]:\r\nruntime.gopark(0x144df51a0, 0x1453daa40, 0xc, 0x14, 0x1)\r\n\t/usr/local/Cellar/go/1.18.3/libexec/src/runtime/proc.go:361 +0xf2 fp=0xc000076768 sp=0xc000076738 pc=0x143d1c892\r\nruntime.goparkunlock(0x1?, 0x0?, 0x0?, 0x0?)\r\n\t/usr/local/Cellar/go/1.18.3/libexec/src/runtime/proc.go:367 +0x2a fp=0xc000076798 sp=0xc000076768 pc=0x143d1c92a\r\nruntime.bgsweep(0x0?)\r\n\t/usr/local/Cellar/go/1.18.3/libexec/src/runtime/mgcsweep.go:297 +0xd1 fp=0xc0000767c8 sp=0xc000076798 pc=0x143d07951\r\nruntime.gcenable.func1()\r\n\t/usr/local/Cellar/go/1.18.3/libexec/src/runtime/mgc.go:177 +0x26 fp=0xc0000767e0 sp=0xc0000767c8 pc=0x143cfd626\r\nruntime.goexit()\r\n\t/usr/local/Cellar/go/1.18.3/libexec/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc0000767e8 sp=0xc0000767e0 pc=0x143d4b501\r\ncreated by runtime.gcenable\r\n\t/usr/local/Cellar/go/1.18.3/libexec/src/runtime/mgc.go:177 +0x6b\r\n\r\ngoroutine 19 [GC scavenge wait]:\r\nruntime.gopark(0x144df51a0, 0x1453daa00, 0xd, 0x14, 0x1)\r\n\t/usr/local/Cellar/go/1.18.3/libexec/src/runtime/proc.go:361 +0xf2 fp=0xc000076ef0 sp=0xc000076ec0 pc=0x143d1c892\r\nruntime.goparkunlock(0x1453f5d28?, 0x0?, 0x0?, 0x0?)\r\n\t/usr/local/Cellar/go/1.18.3/libexec/src/runtime/proc.go:367 +0x2a fp=0xc000076f20 sp=0xc000076ef0 pc=0x143d1c92a\r\nruntime.bgscavenge(0x0?)\r\n\t/usr/local/Cellar/go/1.18.3/libexec/src/runtime/mgcscavenge.go:364 +0x2ca fp=0xc000076fc8 sp=0xc000076f20 pc=0x143d0556a\r\nruntime.gcenable.func2()\r\n\t/usr/local/Cellar/go/1.18.3/libexec/src/runtime/mgc.go:178 +0x26 fp=0xc000076fe0 sp=0xc000076fc8 pc=0x143cfd5c6\r\nruntime.goexit()\r\n\t/usr/local/Cellar/go/1.18.3/libexec/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc000076fe8 sp=0xc000076fe0 pc=0x143d4b501\r\ncreated by runtime.gcenable\r\n\t/usr/local/Cellar/go/1.18.3/libexec/src/runtime/mgc.go:178 +0xaa\r\n\r\ngoroutine 34 [finalizer wait]:\r\nruntime.gopark(0x144df51a0, 0x14540d140, 0x10, 0x14, 0x1)\r\n\t/usr/local/Cellar/go/1.18.3/libexec/src/runtime/proc.go:361 +0xf2 fp=0xc000120e00 sp=0xc000120dd0 pc=0x143d1c892\r\nruntime.goparkunlock(0x0?, 0x60?, 0xd6?, 0xc000136000?)\r\n\t/usr/local/Cellar/go/1.18.3/libexec/src/runtime/proc.go:367 +0x2a fp=0xc000120e30 sp=0xc000120e00 pc=0x143d1c92a\r\nruntime.runfinq()\r\n\t/usr/local/Cellar/go/1.18.3/libexec/src/runtime/mfinal.go:177 +0xab fp=0xc000120fe0 sp=0xc000120e30 pc=0x143cfc6ab\r\nruntime.goexit()\r\n\t/usr/local/Cellar/go/1.18.3/libexec/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc000120fe8 sp=0xc000120fe0 pc=0x143d4b501\r\ncreated by runtime.createfing\r\n\t/usr/local/Cellar/go/1.18.3/libexec/src/runtime/mfinal.go:157 +0x45\r\n\r\ngoroutine 36 [select, locked to thread]:\r\nruntime.gopark(0x144df51f8, 0x0, 0x9, 0x18, 0x1)\r\n\t/usr/local/Cellar/go/1.18.3/libexec/src/runtime/proc.go:361 +0xf2 fp=0xc0003e6df8 sp=0xc0003e6dc8 pc=0x143d1c892\r\nruntime.selectgo(0xc0003e6fa8, 0xc0003e6fa0, 0x0?, 0x0, 0x0?, 0x1)\r\n\t/usr/local/Cellar/go/1.18.3/libexec/src/runtime/select.go:328 +0xa45 fp=0xc0003e6f50 sp=0xc0003e6df8 pc=0x143d2bf65\r\nruntime.ensureSigM.func1()\r\n\t/usr/local/Cellar/go/1.18.3/libexec/src/runtime/signal_unix.go:973 +0x14a fp=0xc0003e6fe0 sp=0xc0003e6f50 pc=0x143d3006a\r\nruntime.goexit()\r\n\t/usr/local/Cellar/go/1.18.3/libexec/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc0003e6fe8 sp=0xc0003e6fe0 pc=0x143d4b501\r\ncreated by runtime.ensureSigM\r\n\t/usr/local/Cellar/go/1.18.3/libexec/src/runtime/signal_unix.go:956 +0xbd\r\n\r\ngoroutine 3 [syscall]:\r\nruntime.sigNoteSleep(0x0)\r\n\t/usr/local/Cellar/go/1.18.3/libexec/src/runtime/os_darwin.go:123 +0x1e fp=0xc00007b790 sp=0xc00007b758 pc=0x143d168fe\r\nos/signal.signal_recv()\r\n\t/usr/local/Cellar/go/1.18.3/libexec/src/runtime/sigqueue.go:148 +0x28 fp=0xc00007b7b0 sp=0xc00007b790 pc=0x143d47968\r\nos/signal.loop()\r\n\t/usr/local/Cellar/go/1.18.3/libexec/src/os/signal/signal_unix.go:23 +0x1d fp=0xc00007b7e0 sp=0xc00007b7b0 pc=0x143ea003d\r\nruntime.goexit()\r\n\t/usr/local/Cellar/go/1.18.3/libexec/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc00007b7e8 sp=0xc00007b7e0 pc=0x143d4b501\r\ncreated by os/signal.Notify.func1.1\r\n\t/usr/local/Cellar/go/1.18.3/libexec/src/os/signal/signal.go:151 +0x2e\r\n\r\ngoroutine 37 [GC worker (idle)]:\r\nruntime.gopark(0x144df5038, 0xc0001b2da0, 0x18, 0x14, 0x0)\r\n\t/usr/local/Cellar/go/1.18.3/libexec/src/runtime/proc.go:361 +0xf2 fp=0xc0003e7750 sp=0xc0003e7720 pc=0x143d1c892\r\nruntime.gcBgMarkWorker()\r\n\t/usr/local/Cellar/go/1.18.3/libexec/src/runtime/mgc.go:1207 +0x107 fp=0xc0003e77e0 sp=0xc0003e7750 pc=0x143cff187\r\nruntime.goexit()\r\n\t/usr/local/Cellar/go/1.18.3/libexec/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc0003e77e8 sp=0xc0003e77e0 pc=0x143d4b501\r\ncreated by runtime.gcBgMarkStartWorkers\r\n\t/usr/local/Cellar/go/1.18.3/libexec/src/runtime/mgc.go:1131 +0x25\r\n\r\ngoroutine 38 [runnable]:\r\nruntime.gcMarkDone()\r\n\t/usr/local/Cellar/go/1.18.3/libexec/src/runtime/mgc.go:775 +0x239 fp=0xc0003e2750 sp=0xc0003e2748 pc=0x143cfe299\r\nruntime.gcBgMarkWorker()\r\n\t/usr/local/Cellar/go/1.18.3/libexec/src/runtime/mgc.go:1331 +0x296 fp=0xc0003e27e0 sp=0xc0003e2750 pc=0x143cff316\r\nruntime.goexit()\r\n\t/usr/local/Cellar/go/1.18.3/libexec/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc0003e27e8 sp=0xc0003e27e0 pc=0x143d4b501\r\ncreated by runtime.gcBgMarkStartWorkers\r\n\t/usr/local/Cellar/go/1.18.3/libexec/src/runtime/mgc.go:1131 +0x25\r\n\r\ngoroutine 23 [GC worker (idle)]:\r\nruntime.gopark(0x144df5038, 0xc00014a2a0, 0x18, 0x14, 0x0)\r\n\t/usr/local/Cellar/go/1.18.3/libexec/src/runtime/proc.go:361 +0xf2 fp=0xc0003e6750 sp=0xc0003e6720 pc=0x143d1c892\r\nruntime.gcBgMarkWorker()\r\n\t/usr/local/Cellar/go/1.18.3/libexec/src/runtime/mgc.go:1207 +0x107 fp=0xc0003e67e0 sp=0xc0003e6750 pc=0x143cff187\r\nruntime.goexit()\r\n\t/usr/local/Cellar/go/1.18.3/libexec/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc0003e67e8 sp=0xc0003e67e0 pc=0x143d4b501\r\ncreated by runtime.gcBgMarkStartWorkers\r\n\t/usr/local/Cellar/go/1.18.3/libexec/src/runtime/mgc.go:1131 +0x25\r\n\r\ngoroutine 52 [GC worker (idle)]:\r\nruntime.gopark(0x144df5038, 0xc0001b2d20, 0x18, 0x14, 0x0)\r\n\t/usr/local/Cellar/go/1.18.3/libexec/src/runtime/proc.go:361 +0xf2 fp=0xc0003e2f50 sp=0xc0003e2f20 pc=0x143d1c892\r\nruntime.gcBgMarkWorker()\r\n\t/usr/local/Cellar/go/1.18.3/libexec/src/runtime/mgc.go:1207 +0x107 fp=0xc0003e2fe0 sp=0xc0003e2f50 pc=0x143cff187\r\nruntime.goexit()\r\n\t/usr/local/Cellar/go/1.18.3/libexec/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc0003e2fe8 sp=0xc0003e2fe0 pc=0x143d4b501\r\ncreated by runtime.gcBgMarkStartWorkers\r\n\t/usr/local/Cellar/go/1.18.3/libexec/src/runtime/mgc.go:1131 +0x25\r\n\r\ngoroutine 67 [GC worker (idle)]:\r\nruntime.gopark(0x144df5038, 0xc00003a0c0, 0x18, 0x14, 0x0)\r\n\t/usr/local/Cellar/go/1.18.3/libexec/src/runtime/proc.go:361 +0xf2 fp=0xc000506750 sp=0xc000506720 pc=0x143d1c892\r\nruntime.gcBgMarkWorker()\r\n\t/usr/local/Cellar/go/1.18.3/libexec/src/runtime/mgc.go:1207 +0x107 fp=0xc0005067e0 sp=0xc000506750 pc=0x143cff187\r\nruntime.goexit()\r\n\t/usr/local/Cellar/go/1.18.3/libexec/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc0005067e8 sp=0xc0005067e0 pc=0x143d4b501\r\ncreated by runtime.gcBgMarkStartWorkers\r\n\t/usr/local/Cellar/go/1.18.3/libexec/src/runtime/mgc.go:1131 +0x25\r\n\r\ngoroutine 4 [GC worker (idle)]:\r\nruntime.gopark(0x144df5038, 0xc0001b2d40, 0x18, 0x14, 0x0)\r\n\t/usr/local/Cellar/go/1.18.3/libexec/src/runtime/proc.go:361 +0xf2 fp=0xc00007bf50 sp=0xc00007bf20 pc=0x143d1c892\r\nruntime.gcBgMarkWorker()\r\n\t/usr/local/Cellar/go/1.18.3/libexec/src/runtime/mgc.go:1207 +0x107 fp=0xc00007bfe0 sp=0xc00007bf50 pc=0x143cff187\r\nruntime.goexit()\r\n\t/usr/local/Cellar/go/1.18.3/libexec/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc00007bfe8 sp=0xc00007bfe0 pc=0x143d4b501\r\ncreated by runtime.gcBgMarkStartWorkers\r\n\t/usr/local/Cellar/go/1.18.3/libexec/src/runtime/mgc.go:1131 +0x25\r\n\r\ngoroutine 68 [GC worker (idle)]:\r\nruntime.gopark(0x144df5038, 0xc0001b2d60, 0x18, 0x14, 0x0)\r\n\t/usr/local/Cellar/go/1.18.3/libexec/src/runtime/proc.go:361 +0xf2 fp=0xc000506f50 sp=0xc000506f20 pc=0x143d1c892\r\nruntime.gcBgMarkWorker()\r\n\t/usr/local/Cellar/go/1.18.3/libexec/src/runtime/mgc.go:1207 +0x107 fp=0xc000506fe0 sp=0xc000506f50 pc=0x143cff187\r\nruntime.goexit()\r\n\t/usr/local/Cellar/go/1.18.3/libexec/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc000506fe8 sp=0xc000506fe0 pc=0x143d4b501\r\ncreated by runtime.gcBgMarkStartWorkers\r\n\t/usr/local/Cellar/go/1.18.3/libexec/src/runtime/mgc.go:1131 +0x25\r\n\r\ngoroutine 5 [GC worker (idle)]:\r\nruntime.gopark(0x144df5038, 0xc0001b2d80, 0x18, 0x14, 0x0)\r\n\t/usr/local/Cellar/go/1.18.3/libexec/src/runtime/proc.go:361 +0xf2 fp=0xc00007c750 sp=0xc00007c720 pc=0x143d1c892\r\nruntime.gcBgMarkWorker()\r\n\t/usr/local/Cellar/go/1.18.3/libexec/src/runtime/mgc.go:1207 +0x107 fp=0xc00007c7e0 sp=0xc00007c750 pc=0x143cff187\r\nruntime.goexit()\r\n\t/usr/local/Cellar/go/1.18.3/libexec/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc00007c7e8 sp=0xc00007c7e0 pc=0x143d4b501\r\ncreated by runtime.gcBgMarkStartWorkers\r\n\t/usr/local/Cellar/go/1.18.3/libexec/src/runtime/mgc.go:1131 +0x25\r\n\r\ngoroutine 6 [GC worker (idle)]:\r\nruntime.gopark(0x144df5038, 0xc0001b2dc0, 0x18, 0x14, 0x0)\r\n\t/usr/local/Cellar/go/1.18.3/libexec/src/runtime/proc.go:361 +0xf2 fp=0xc00007cf50 sp=0xc00007cf20 pc=0x143d1c892\r\nruntime.gcBgMarkWorker()\r\n\t/usr/local/Cellar/go/1.18.3/libexec/src/runtime/mgc.go:1207 +0x107 fp=0xc00007cfe0 sp=0xc00007cf50 pc=0x143cff187\r\nruntime.goexit()\r\n\t/usr/local/Cellar/go/1.18.3/libexec/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc00007cfe8 sp=0xc00007cfe0 pc=0x143d4b501\r\ncreated by runtime.gcBgMarkStartWorkers\r\n\t/usr/local/Cellar/go/1.18.3/libexec/src/runtime/mgc.go:1131 +0x25\r\n\r\ngoroutine 7 [GC worker (idle)]:\r\nruntime.gopark(0x144df5038, 0xc0001b2de0, 0x18, 0x14, 0x0)\r\n\t/usr/local/Cellar/go/1.18.3/libexec/src/runtime/proc.go:361 +0xf2 fp=0xc00007d750 sp=0xc00007d720 pc=0x143d1c892\r\nruntime.gcBgMarkWorker()\r\n\t/usr/local/Cellar/go/1.18.3/libexec/src/runtime/mgc.go:1207 +0x107 fp=0xc00007d7e0 sp=0xc00007d750 pc=0x143cff187\r\nruntime.goexit()\r\n\t/usr/local/Cellar/go/1.18.3/libexec/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc00007d7e8 sp=0xc00007d7e0 pc=0x143d4b501\r\ncreated by runtime.gcBgMarkStartWorkers\r\n\t/usr/local/Cellar/go/1.18.3/libexec/src/runtime/mgc.go:1131 +0x25\r\n\r\ngoroutine 69 [GC worker (idle)]:\r\nruntime.gopark(0x144df5038, 0xc0001b2e00, 0x18, 0x14, 0x0)\r\n\t/usr/local/Cellar/go/1.18.3/libexec/src/runtime/proc.go:361 +0xf2 fp=0xc000507750 sp=0xc000507720 pc=0x143d1c892\r\nruntime.gcBgMarkWorker()\r\n\t/usr/local/Cellar/go/1.18.3/libexec/src/runtime/mgc.go:1207 +0x107 fp=0xc0005077e0 sp=0xc000507750 pc=0x143cff187\r\nruntime.goexit()\r\n\t/usr/local/Cellar/go/1.18.3/libexec/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc0005077e8 sp=0xc0005077e0 pc=0x143d4b501\r\ncreated by runtime.gcBgMarkStartWorkers\r\n\t/usr/local/Cellar/go/1.18.3/libexec/src/runtime/mgc.go:1131 +0x25\r\n\r\ngoroutine 70 [GC worker (idle)]:\r\nruntime.gopark(0x144df5038, 0xc0001b2e20, 0x18, 0x14, 0x0)\r\n\t/usr/local/Cellar/go/1.18.3/libexec/src/runtime/proc.go:361 +0xf2 fp=0xc000507f50 sp=0xc000507f20 pc=0x143d1c892\r\nruntime.gcBgMarkWorker()\r\n\t/usr/local/Cellar/go/1.18.3/libexec/src/runtime/mgc.go:1207 +0x107 fp=0xc000507fe0 sp=0xc000507f50 pc=0x143cff187\r\nruntime.goexit()\r\n\t/usr/local/Cellar/go/1.18.3/libexec/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc000507fe8 sp=0xc000507fe0 pc=0x143d4b501\r\ncreated by runtime.gcBgMarkStartWorkers\r\n\t/usr/local/Cellar/go/1.18.3/libexec/src/runtime/mgc.go:1131 +0x25\r\n\r\ngoroutine 39 [GC worker (idle)]:\r\nruntime.gopark(0x144df5038, 0xc00003a0e0, 0x18, 0x14, 0x0)\r\n\t/usr/local/Cellar/go/1.18.3/libexec/src/runtime/proc.go:361 +0xf2 fp=0xc0003e7f50 sp=0xc0003e7f20 pc=0x143d1c892\r\nruntime.gcBgMarkWorker()\r\n\t/usr/local/Cellar/go/1.18.3/libexec/src/runtime/mgc.go:1207 +0x107 fp=0xc0003e7fe0 sp=0xc0003e7f50 pc=0x143cff187\r\nruntime.goexit()\r\n\t/usr/local/Cellar/go/1.18.3/libexec/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc0003e7fe8 sp=0xc0003e7fe0 pc=0x143d4b501\r\ncreated by runtime.gcBgMarkStartWorkers\r\n\t/usr/local/Cellar/go/1.18.3/libexec/src/runtime/mgc.go:1131 +0x25\r\n\r\ngoroutine 71 [GC worker (idle)]:\r\nruntime.gopark(0x144df5038, 0xc00003a100, 0x18, 0x14, 0x0)\r\n\t/usr/local/Cellar/go/1.18.3/libexec/src/runtime/proc.go:361 +0xf2 fp=0xc000508750 sp=0xc000508720 pc=0x143d1c892\r\nruntime.gcBgMarkWorker()\r\n\t/usr/local/Cellar/go/1.18.3/libexec/src/runtime/mgc.go:1207 +0x107 fp=0xc0005087e0 sp=0xc000508750 pc=0x143cff187\r\nruntime.goexit()\r\n\t/usr/local/Cellar/go/1.18.3/libexec/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc0005087e8 sp=0xc0005087e0 pc=0x143d4b501\r\ncreated by runtime.gcBgMarkStartWorkers\r\n\t/usr/local/Cellar/go/1.18.3/libexec/src/runtime/mgc.go:1131 +0x25\r\n\r\ngoroutine 8 [GC worker (idle)]:\r\nruntime.gopark(0x144df5038, 0xc0001b2e40, 0x18, 0x14, 0x0)\r\n\t/usr/local/Cellar/go/1.18.3/libexec/src/runtime/proc.go:361 +0xf2 fp=0xc00007df50 sp=0xc00007df20 pc=0x143d1c892\r\nruntime.gcBgMarkWorker()\r\n\t/usr/local/Cellar/go/1.18.3/libexec/src/runtime/mgc.go:1207 +0x107 fp=0xc00007dfe0 sp=0xc00007df50 pc=0x143cff187\r\nruntime.goexit()\r\n\t/usr/local/Cellar/go/1.18.3/libexec/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc00007dfe8 sp=0xc00007dfe0 pc=0x143d4b501\r\ncreated by runtime.gcBgMarkStartWorkers\r\n\t/usr/local/Cellar/go/1.18.3/libexec/src/runtime/mgc.go:1131 +0x25\r\n\r\ngoroutine 72 [GC worker (idle)]:\r\nruntime.gopark(0x144df5038, 0xc0001b2e60, 0x18, 0x14, 0x0)\r\n\t/usr/local/Cellar/go/1.18.3/libexec/src/runtime/proc.go:361 +0xf2 fp=0xc000508f50 sp=0xc000508f20 pc=0x143d1c892\r\nruntime.gcBgMarkWorker()\r\n\t/usr/local/Cellar/go/1.18.3/libexec/src/runtime/mgc.go:1207 +0x107 fp=0xc000508fe0 sp=0xc000508f50 pc=0x143cff187\r\nruntime.goexit()\r\n\t/usr/local/Cellar/go/1.18.3/libexec/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc000508fe8 sp=0xc000508fe0 pc=0x143d4b501\r\ncreated by runtime.gcBgMarkStartWorkers\r\n\t/usr/local/Cellar/go/1.18.3/libexec/src/runtime/mgc.go:1131 +0x25\r\n/Users/achal/.pyenv/versions/3.8.12/lib/python3.8/multiprocessing/resource_tracker.py:216: UserWarning: resource_tracker: There appear to be 1 leaked semaphore objects to clean up at shutdown\r\n  warnings.warn('resource_tracker: There appear to be %d '\r\n```\r\n\r\nThe most suspicious segment of this trace is here:\r\n```\r\nsync.(*Map).Store(0x1453da440, {0x144ca1480, 0xc00062ce20}, {0x144d763a0, 0xc00059e840})\r\n\t/usr/local/Cellar/go/1.18.3/libexec/src/sync/map.go:137 +0x91 fp=0xc00071af28 sp=0xc00071ae30 pc=0x143d4fc51\r\ngithub.com/apache/arrow/go/v8/arrow/cdata.storeData({0x144e032d8, 0xc00059e840})\r\n\t/Users/achal/go/pkg/mod/github.com/apache/arrow/go/v8@v8.0.0-20220408212425-58fe60f59289/arrow/cdata/exports.go:45 +0xc5 fp=0xc00071af90 sp=0xc00071af28 pc=0x1441674e5\r\ngithub.com/apache/arrow/go/v8/arrow/cdata.exportArray({0x144e044c0, 0xc0005ad6c0}, 0x7fd7443c0990, 0x0)\r\n\t/Users/achal/go/pkg/mod/github.com/apache/arrow/go/v8@v8.0.0-20220408212425-58fe60f59289/arrow/cdata/cdata_exports.go:356 +0x365 fp=0xc00071b260 sp=0xc00071af90 pc=0x144166945\r\ngithub.com/apache/arrow/go/v8/arrow/cdata.exportArray({0x144e04800, 0xc0005ad540}, 0x6000025299b0, 0x0)\r\n\t/Users/achal/go/pkg/mod/github.com/apache/arrow/go/v8@v8.0.0-20220408212425-58fe60f59289/arrow/cdata/cdata_exports.go:385 +0xae7 fp=0xc00071b530 sp=0xc00071b260 pc=0x1441670c7\r\ngithub.com/apache/arrow/go/v8/arrow/cdata.ExportArrowRecordBatch({0x144e04b40, 0xc00081fc20}, 0x6000025299b0, 0x6000025298b0)\r\n\t/Users/achal/go/pkg/mod/github.com/apache/arrow/go/v8@v8.0.0-20220408212425-58fe60f59289/arrow/cdata/interface.go:217 +0x413 fp=0xc00071b740 sp=0xc00071b530 pc=0x14415c433\r\ngithub.com/feast-dev/feast/go/embedded.(*OnlineFeatureService).GetOnlineFeatures(0xc0004b0000, {0xc0000b8c00, 0xa, 0x10}, {0x0, 0x0}, {0x600002529a30, 0x600002528530}, {0x600002529db0, 0x60000252a430}, ...)\r\n\t/Users/achal/tecton/feast/go/embedded/online_features.go:220 +0xef0 fp=0xc00071bbc8 sp=0xc00071b740 pc=0x1449fa470\r\nmain.embedded_OnlineFeatureService_GetOnlineFeatures(0x2, 0x3fd, 0x10d233320, 0x3fe, 0x3ff, 0x1, 0x400)\r\n\t/Users/achal/tecton/feast/build/lib.macosx-12.0-x86_64-3.8/feast/embedded_go/lib/embedded.go:1350 +0x365 fp=0xc00071bda0 sp=0xc00071bbc8 pc=0x144a063e5\r\n_cgoexp_79ecca462223_embedded_OnlineFeatureService_GetOnlineFeatures(0x7ff7b3210608)\r\n\t_cgo_gotypes.go:1936 +0xcd fp=0xc00071be38 sp=0xc00071bda0 pc=0x144a0ec6d\r\n```\r\n\r\nWhich makes me suspect that there's either a bug in arrow somewhere? \r\n\r\n## Steps to reproduce\r\n\r\nI'll push my changes to a branch.\r\n\r\n## Possible Solution\r\n\r\n?\r\n","author":{"url":"https://github.com/achals","@type":"Person","name":"achals"},"datePublished":"2022-06-23T22:06:46.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":3},"url":"https://github.com/2849/feast/issues/2849"}