René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"To cross build, or not to cross build Ingestion","articleBody":"In #451 we upgraded outright to Java 11 for master / Feast v0.5+. Subsequently in #473 we restored support for v0.3 and v0.4 maintenance branches to be tested on CI with Java 8 (thanks!). I believe [that has been broken][1] by later updates to CI scripts, I'll consider that part of this issue.\r\n\r\nThe upgrade could pose problems for Feast's Beam-based Ingestion component. Beam itself [does not build with Java 11 yet][2], and the scope of that issue is only the DirectRunner. Other runners could lag further behind, as for example Spark does not build or run on Java 11 until its version 3.0, which is in preview. Thus I presume Beam's SparkRunner will not build with Java 11 until a GM Spark release can.\r\n\r\nFeast does not support any runners other than GC Dataflow (and DirectRunner), so there may be no immediate issue if these are working so far. But, supporting additional runners—if anticipated (#362)—could be blocked if Ingestion does not remain Java 8-compatible.\r\n\r\nI wanted to raise the question of whether we should (cross?) build Ingestion with Java 8 to ensure it remains compatible for awhile longer.\r\n\r\nThe bug report that follows applies to current status quo, regardless of the question above.\r\n\r\n[1]: https://github.com/gojek/feast/pull/516#issuecomment-596076642\r\n[2]: https://issues.apache.org/jira/browse/BEAM-2530\r\n\r\n## Expected Behavior \r\n\r\n1. Feast master gives a helpful error about Java requirement if a developer attempts to build with an older version than is supported.\r\n1. Feast v0.3-branch and v0.4-branch can build \u0026 test on CI with Java 8. (Appears fixed by #523 🎉)\r\n\r\n## Current Behavior\r\n\r\n    $ jenv version\r\n    openjdk64-1.8.0.242 (set by /Users/cmartin/.jenv/version)\r\n    $ mvn compile\r\n    \u003cSNIP\u003e\r\n    [ERROR] Fatal error compiling: invalid flag: --release\r\n\r\nThis stems from:\r\n\r\nhttps://github.com/gojek/feast/blob/7633912cd8209cece2a21ebe344600342ef1ff9b/pom.xml#L408\r\n\r\nbecause the flag was added to `javac` in Java 9. Maven Enforcer will catch this, proposed below.\r\n\r\nFor the second problem, CI fails with:\r\n\r\n```\r\n.prow/scripts/test-end-to-end-batch.sh only supports Debian stretch.\r\nPlease change your operating system to use this script.\r\n```\r\n\r\n## Steps to reproduce\r\n\r\n1. As above demo\r\n1. Submit a PR to v0.3-branch as in #516 \r\n\r\n## Possible Solution\r\n\r\nThe unhelpful error from Maven would be improved by reverting 190e605, resulting in output like\r\n\r\n```\r\n[WARNING] Rule 1: org.apache.maven.plugins.enforcer.RequireJavaVersion failed with message:\r\nDetected JDK Version: 1.8.0-242 is not in the allowed range [11.0,).\r\n\r\n[ERROR] Some Enforcer rules have failed. Look above for specific messages explaining why the rule failed.\r\n```\r\n\r\nFor the CI environment problem, restore compatibility of the test scripts with `maven:3.6-jdk-8` Docker image. Prow always builds using master versions of scripts \u0026 config, I think? Ideally then, ensure a build is run on the maintenance branches when Prow scripts are changed, for as long as those Feast versions are supported.\r\n\r\nFor the Java 8 matter, a quick test seems to tell me that maven-compiler-plugin can compile select modules in a Reactor build with `\u003crelease\u003e8\u003c/release\u003e` while the others compile with 11. So I'm suggesting that it could be straightforward to _only_ build Ingestion for 8, instead of cross-building for two versions and having a more complicated CI matrix. Note then that `datatypes-java` ought to also build with 8 to be safe.","author":{"url":"https://github.com/ches","@type":"Person","name":"ches"},"datePublished":"2020-03-07T15:55:52.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":8},"url":"https://github.com/517/feast/issues/517"}