René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Show risky disabled tests as Problems and in SARIF-report","articleBody":"**Description**\r\n\r\nGenerated tests disabled due to Sandbox or due to possible JVM crash are not show in SARIF-report and on Problems tab.\r\n\r\n**To Reproduce**\r\n\r\n1. Run the 'UTBotJava' project in IntelliJ Idea\r\n2. Both Fuzzing and Symbolic analysis are enabled\r\n3. UnitTestBot plugin is installed from branch #1166: [build](https://github.com/UnitTestBot/UTBotJava/actions/runs/3264184181)\r\n4. In Project view find utbot-sample/src/main/java/org/utbot/examples/exceptions/JvmCrashExamples.java\r\n5. Generate tests with UnitTestBot...\r\n6. Check there are tests \"Disabled due to possible JVM crash\" and \"Disabled due to sandbox\"\r\n\r\n**Expected behavior**\r\n\r\nThese found problems need to:\r\n1) add to error suite\r\n2) add to SARIF-report \r\n3) display on Problems tab with the respectful stack trace\r\n\r\n**Actual behavior**\r\n\r\nCurrently executions that are stopped on AccessControlException - Disabled due to sandbox - are not added to SARIF report as errors and are not displayed on Problems tab.\r\nSame with the tests \"Disabled due to possible JVM crash\".\r\n\r\n**Visual proofs (screenshots, logs, images)**\r\n\r\n![image](https://user-images.githubusercontent.com/37301492/196181527-37fcda27-7053-4823-8182-e7bb31143482.png)\r\n\r\n![image](https://user-images.githubusercontent.com/37301492/196181647-7b041f65-1532-41f1-a799-84440f03a3db.png)\r\n\r\n```java\r\npublic class JvmCrashExamplesTest {\r\n    ///region Test suites for executable org.utbot.examples.exceptions.JvmCrashExamples.crash\r\n\r\n    ///region OTHER: SECURITY for method crash(int)\r\n\r\n    @Test\r\n    @org.junit.Ignore(value = \"Disabled due to sandbox\")\r\n    public void testCrash1() {\r\n        JvmCrashExamples jvmCrashExamples = new JvmCrashExamples();\r\n        \r\n        /* This test fails because method [org.utbot.examples.exceptions.JvmCrashExamples.crash] produces [java.security.AccessControlException: access denied (\"java.lang.RuntimePermission\" \"accessClassInPackage.sun.misc\")]\r\n            java.base/java.security.AccessControlContext.checkPermission(AccessControlContext.java:472)\r\n            java.base/java.security.AccessController.checkPermission(AccessController.java:897)\r\n            java.base/java.lang.SecurityManager.checkPermission(SecurityManager.java:322)\r\n            java.base/java.lang.SecurityManager.checkPackageAccess(SecurityManager.java:1290)\r\n            java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:174)\r\n            java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:576)\r\n            org.utbot.instrumentation.process.HandlerClassesLoader.loadClass(ChildProcess.kt:48)\r\n            java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522)\r\n            org.utbot.examples.exceptions.JvmCrashExamples.crash(JvmCrashExamples.java:20) */\r\n    }\r\n    ///endregion\r\n\r\n    ///endregion\r\n\r\n    ///region Test suites for executable org.utbot.examples.exceptions.JvmCrashExamples.crashPrivileged\r\n\r\n    ///region\r\n\r\n    @Test\r\n    @org.junit.Ignore(value = \"Disabled due to possible JVM crash\")\r\n    public void testCrashPrivileged1() {\r\n        JvmCrashExamples jvmCrashExamples = new JvmCrashExamples();\r\n\r\n        // This invocation possibly crashes JVM\r\n        jvmCrashExamples.crashPrivileged(1);\r\n    }\r\n    ///endregion\r\n\r\n    ///endregion\r\n\r\n    ///region Test suites for executable org.utbot.examples.exceptions.JvmCrashExamples.exit\r\n\r\n    ///region FUZZER: SUCCESSFUL EXECUTIONS for method exit(int)\r\n\r\n    /**\r\n     * @utbot.classUnderTest {@link JvmCrashExamples}\r\n     * @utbot.methodUnderTest {@link JvmCrashExamples#exit(int)}\r\n     */\r\n    @Test\r\n    public void testExitReturnsOne() {\r\n        JvmCrashExamples jvmCrashExamples = new JvmCrashExamples();\r\n\r\n        int actual = jvmCrashExamples.exit(1);\r\n\r\n        assertEquals(1, actual);\r\n    }\r\n    ///endregion\r\n\r\n    ///region OTHER: SECURITY for method exit(int)\r\n\r\n    @Test\r\n    @org.junit.Ignore(value = \"Disabled due to sandbox\")\r\n    public void testExit1() {\r\n        JvmCrashExamples jvmCrashExamples = new JvmCrashExamples();\r\n        \r\n        /* This test fails because method [org.utbot.examples.exceptions.JvmCrashExamples.exit] produces [java.security.AccessControlException: access denied (\"java.lang.RuntimePermission\" \"exitVM.-1\")]\r\n            java.base/java.security.AccessControlContext.checkPermission(AccessControlContext.java:472)\r\n            java.base/java.security.AccessController.checkPermission(AccessController.java:897)\r\n            java.base/java.lang.SecurityManager.checkPermission(SecurityManager.java:322)\r\n            java.base/java.lang.SecurityManager.checkExit(SecurityManager.java:534)\r\n            java.base/java.lang.Runtime.exit(Runtime.java:114)\r\n            java.base/java.lang.System.exit(System.java:1752)\r\n            org.utbot.examples.exceptions.JvmCrashExamples.exit(JvmCrashExamples.java:12) */\r\n    }\r\n    ///endregion\r\n\r\n    ///endregion\r\n}\r\n\r\n```\r\n\r\n**Additional context**\r\n\r\n_Originally posted by @alisevych in https://github.com/UnitTestBot/UTBotJava/issues/1166#issuecomment-1280800929_\r\n      ","author":{"url":"https://github.com/alisevych","@type":"Person","name":"alisevych"},"datePublished":"2022-10-17T12:55:09.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":1},"url":"https://github.com/1168/UTBotJava/issues/1168"}