René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"k8s launcher python version compatibility (Python 3.7/3.8)","articleBody":"Running historical retrieval with `os.environ[\"FEAST_SPARK_LAUNCHER\"] = \"k8s\"` breaks due to python compatibility with generated spark python file:\r\n\r\n```\r\nkubectl logs feast-m5mz27p6-driver -n spark-operator                                                                            main@d2a800f ✘\r\n++ id -u\r\n+ myuid=0\r\n++ id -g\r\n+ mygid=0\r\n+ set +e\r\n++ getent passwd 0\r\n+ uidentry=root:x:0:0:root:/root:/bin/bash\r\n+ set -e\r\n+ '[' -z root:x:0:0:root:/root:/bin/bash ']'\r\n+ SPARK_CLASSPATH=':/opt/spark/jars/*'\r\n+ env\r\n+ grep SPARK_JAVA_OPT_\r\n+ sort -t_ -k4 -n\r\n+ sed 's/[^=]*=\\(.*\\)/\\1/g'\r\n+ readarray -t SPARK_EXECUTOR_JAVA_OPTS\r\n+ '[' -n '' ']'\r\n+ '[' '' == 2 ']'\r\n+ '[' '' == 3 ']'\r\n+ '[' -n '' ']'\r\n+ '[' -z ']'\r\n+ case \"$1\" in\r\n+ shift 1\r\n+ CMD=(\"$SPARK_HOME/bin/spark-submit\" --conf \"spark.driver.bindAddress=$SPARK_DRIVER_BIND_ADDRESS\" --deploy-mode client \"$@\")\r\n+ exec /usr/bin/tini -s -- /opt/spark/bin/spark-submit --conf spark.driver.bindAddress=10.244.1.33 --deploy-mode client --properties-file /opt/spark/conf/spark.properties --class org.apache.spark.deploy.PythonRunner wasbs://feasttest@feastdrivingpoc.blob.core.windows.net/artifacts/44d600e498692f5f4d9e183718f12fbd7925c1b97205373c181486ccab33f667.py --feature-tables W3siZmVhdHVyZXMiOiBbeyJuYW1lIjogImF2Z19kYWlseV90cmlwcyIsICJ0eXBlIjogIklOVDMyIn0sIHsibmFtZSI6ICJhY2NfcmF0ZSIsICJ0eXBlIjogIkZMT0FUIn0sIHsibmFtZSI6ICJjb252X3JhdGUiLCAidHlwZSI6ICJGTE9BVCJ9XSwgInByb2plY3QiOiAiZmVhc3Rkcml2aW5ncG9jdGVzdCIsICJuYW1lIjogImRyaXZlcl9zdGF0aXN0aWNzIiwgImVudGl0aWVzIjogW3sibmFtZSI6ICJkcml2ZXJfaWQiLCAidHlwZSI6ICJJTlQ2NCJ9XSwgIm1heF9hZ2UiOiA4NjQwMCwgImxhYmVscyI6IHt9fSwgeyJmZWF0dXJlcyI6IFt7Im5hbWUiOiAidHJpcHNfdG9kYXkiLCAidHlwZSI6ICJJTlQzMiJ9XSwgInByb2plY3QiOiAiZmVhc3Rkcml2aW5ncG9jdGVzdCIsICJuYW1lIjogImRyaXZlcl90cmlwcyIsICJlbnRpdGllcyI6IFt7Im5hbWUiOiAiZHJpdmVyX2lkIiwgInR5cGUiOiAiSU5UNjQifV0sICJtYXhfYWdlIjogODY0MDAsICJsYWJlbHMiOiB7fX1d --feature-tables-sources W3siZmlsZSI6IHsiZmllbGRfbWFwcGluZyI6IHt9LCAiZXZlbnRfdGltZXN0YW1wX2NvbHVtbiI6ICJkYXRldGltZSIsICJjcmVhdGVkX3RpbWVzdGFtcF9jb2x1bW4iOiAiY3JlYXRlZCIsICJkYXRlX3BhcnRpdGlvbl9jb2x1bW4iOiAiZGF0ZSIsICJwYXRoIjogIndhc2JzOi8vZmVhc3R0ZXN0QGZlYXN0ZHJpdmluZ3BvYy5ibG9iLmNvcmUud2luZG93cy5uZXQvZHJpdmVyX3N0YXRpc3RpY3MiLCAiZm9ybWF0IjogeyJqc29uX2NsYXNzIjogIlBhcnF1ZXRGb3JtYXQifX19LCB7ImZpbGUiOiB7ImZpZWxkX21hcHBpbmciOiB7fSwgImV2ZW50X3RpbWVzdGFtcF9jb2x1bW4iOiAiZGF0ZXRpbWUiLCAiY3JlYXRlZF90aW1lc3RhbXBfY29sdW1uIjogImNyZWF0ZWQiLCAiZGF0ZV9wYXJ0aXRpb25fY29sdW1uIjogImRhdGUiLCAicGF0aCI6ICJ3YXNiczovL2ZlYXN0dGVzdEBmZWFzdGRyaXZpbmdwb2MuYmxvYi5jb3JlLndpbmRvd3MubmV0L2RyaXZlcl90cmlwcyIsICJmb3JtYXQiOiB7Impzb25fY2xhc3MiOiAiUGFycXVldEZvcm1hdCJ9fX1d --entity-source eyJmaWxlIjogeyJmaWVsZF9tYXBwaW5nIjoge30sICJldmVudF90aW1lc3RhbXBfY29sdW1uIjogImV2ZW50X3RpbWVzdGFtcCIsICJjcmVhdGVkX3RpbWVzdGFtcF9jb2x1bW4iOiAiIiwgImRhdGVfcGFydGl0aW9uX2NvbHVtbiI6ICIiLCAicGF0aCI6ICJ3YXNiczovL2ZlYXN0dGVzdEBmZWFzdGRyaXZpbmdwb2MuYmxvYi5jb3JlLndpbmRvd3MubmV0L2FydGlmYWN0cy9hODYxMTg0My0yY2QxLTQ1MjktYTliNi1mMTEwNGExYTEzYjMiLCAiZm9ybWF0IjogeyJqc29uX2NsYXNzIjogIlBhcnF1ZXRGb3JtYXQifX19 --destination eyJmb3JtYXQiOiAicGFycXVldCIsICJwYXRoIjogImFiZnNzOi8vZmVhc3Rwb2NAZmVhc3Rwb2MuZGZzLmNvcmUud2luZG93cy5uZXQvZmVhc3Qvb3V0LzQ5ODQ1MzRhLWJiOGUtNDVmZi1hYzc3LTM2YzE3ZjA2YjRhOCJ9\r\n22/01/23 22:43:01 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable\r\n22/01/23 22:43:02 WARN MetricsConfig: Cannot locate configuration: tried hadoop-metrics2-azure-file-system.properties,hadoop-metrics2.properties\r\n  File \"/tmp/spark-ac4d3d12-c430-4f11-987a-7a193b8fef54/44d600e498692f5f4d9e183718f12fbd7925c1b97205373c181486ccab33f667.py\", line 73\r\n    event_timestamp_column: str,\r\n                          ^\r\nSyntaxError: invalid syntax\r\nlog4j:WARN No appenders could be found for logger (org.apache.spark.util.ShutdownHookManager).\r\nlog4j:WARN Please initialize the log4j system properly.\r\nlog4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.\r\n```\r\n\r\nUpgrading to python3.8 from source in `gcr.io/kf-feast/spark-py:v3.0.1` doesn't appear to fix the issue either.\r\nUsing historical features example:\r\nhttps://github.com/Azure/feast-azure/blob/main/cluster/samples/feature_store_azure.ipynb","author":{"url":"https://github.com/andrijaperovic","@type":"Person","name":"andrijaperovic"},"datePublished":"2022-01-25T02:13:51.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":0},"url":"https://github.com/105/feast-spark/issues/105"}