Title: Prefer symbolic executions during minimization by dtim · Pull Request #854 · UnitTestBot/UTBotJava · GitHub
Open Graph Title: Prefer symbolic executions during minimization by dtim · Pull Request #854 · UnitTestBot/UTBotJava
X Title: Prefer symbolic executions during minimization by dtim · Pull Request #854 · UnitTestBot/UTBotJava
Description: Description
Suppose we have two executions with the same coverage, one produced by the symbolic engine, and one produced by the fuzzer. From the user's point of view, they are equivalent, and the minimizer can keep any of them. Usually it will be fuzzed one, because fuzzer produces executions earlier that the symbolic engine, and the minimizer orders executions with the same coverage by their order in the execution list.
When parametrized tests are generated, fuzzed tests tend to be excluded due to mock processing issues. If a fuzzed tests has been selected by the minimizer instead of the symbolic one, the corresponding path remains uncovered: one (symbolic) test has been dropped by the minimizer, and another (fuzzed) test has been excluded by the code generator.
The coverage of the entire parametrized test suite may be improved is minimizer selects symbolic executions when possible. It does not guarantee the full coverage (it is possible to have only fuzzed tests for some paths), but we expect that the coverage may be improved that way.
To let users to control the minimizer behavior, new UtSettings option is introduced: preferSymbolicExecutionsDuringMinimization. When it is set to true (default), the minimizer will consider the execution source (specifically, its class: UtSymbolicExecution or UtFuzzedExecution (UtFailedExecution and any potential other direct UtExecution subclasses are assigned the lowest priority, but UtFailedExecution does not have any coverage anyway, so it is processed separately).
Fixes #843
Type of Change
Breaking change (fix or feature that would cause existing functionality to not work as expected)
This fix changes the set of generated tests. It should not decrease coverage, and is expected to increase coverage when generating parametrized tests (please report a bug if coverage is decreased when preferSymbolicExecutionsDuringMinimization is true).
This PR also changes the minimizer interface.
How Has This Been Tested?
Automated Testing
All existing minimizer tests should pass.
Two new minimizer tests have been added:
org.utbot.framework.minimization.MinimizationGreedyEssentialTest#testWithSourcePriority
org.utbot.framework.minimization.MinimizationGreedyEssentialTest#testWithoutSourcePriority
Manual Scenario
Make sure that fuzzer is enabled in UtSettings, and both fuzzer and symbolic engine are enabled in the plugin configuration window.
Create any method simple enough so both fuzzer and symbolic engine can cover all branches. An example:
package simple;
public class SimpleExample {
public int foo(int a, int b) {
if (a > b)
return a - b;
else
return a + b;
}
}
Set UtSettings.preferSymbolicExecutionsDuringMinimization to true and generate a test suite for foo. Test cases produced by the symbolic engine should be generated.
package simple;
import org.junit.Test;
import static org.junit.Assert.assertEquals;
public class SimpleExampleTest {
///region Test suites for executable simple.SimpleExample.foo
///region SUCCESSFUL EXECUTIONS for method foo(int, int)
/**
*
* Test executes conditions:
* {@code (a > b): False }
* returns from: {@code return a + b; }
*
*/
@Test
public void testFoo_ALessOrEqualB() {
SimpleExample simpleExample = new SimpleExample();
int actual = simpleExample.foo(-240, -240);
assertEquals(-480, actual);
}
/**
*
* Test executes conditions:
* {@code (a > b): True }
* returns from: {@code return a - b; }
*
*/
@Test
public void testFoo_AGreaterThanB() {
SimpleExample simpleExample = new SimpleExample();
int actual = simpleExample.foo(1, 0);
assertEquals(1, actual);
}
///endregion
///endregion
}
Set UtSettings.preferSymbolicExecutionsDuringMinimization to false and generate a test suite for foo again. This time, the plugin should generate test cases produced by the fuzzer.
package simple;
import org.junit.Test;
import static org.junit.Assert.assertEquals;
public class SimpleExampleTest {
///region Test suites for executable simple.SimpleExample.foo
///region
@Test
public void testFooReturnsZero() {
SimpleExample simpleExample = new SimpleExample();
int actual = simpleExample.foo(-1, 1);
assertEquals(0, actual);
}
@Test
public void testFooWithCornerCase() {
SimpleExample simpleExample = new SimpleExample();
int actual = simpleExample.foo(Integer.MAX_VALUE, 1);
assertEquals(2147483646, actual);
}
///endregion
///endregion
}
Checklist (remove irrelevant options):
This is the author self-check list
The change followed the style guidelines of the UTBot project
Self-review of the code is passed
The change contains enough commentaries, particularly in hard-to-understand areas
New documentation is provided or existed one is altered
No new warnings
New tests have been added
All tests pass locally with my changes
Open Graph Description: Description Suppose we have two executions with the same coverage, one produced by the symbolic engine, and one produced by the fuzzer. From the user's point of view, they are equivalent, and t...
X Description: Description Suppose we have two executions with the same coverage, one produced by the symbolic engine, and one produced by the fuzzer. From the user's point of view, they are equivalent, a...
Opengraph URL: https://github.com/UnitTestBot/UTBotJava/pull/854
X: @github
Domain: github.com
| route-pattern | /:user_id/:repository/pull/:id/files(.:format) |
| route-controller | pull_requests |
| route-action | files |
| fetch-nonce | v2:e20c6942-591f-980b-dd69-005b9c494b36 |
| current-catalog-service-hash | ae870bc5e265a340912cde392f23dad3671a0a881730ffdadd82f2f57d81641b |
| request-id | DFA8:E2AFA:15ECA03:1BE405C:69917E98 |
| html-safe-nonce | ed6de0b3cdfea6592cb71438881e946f1004c787ab11d8a88fcb2f8395582bb6 |
| visitor-payload | eyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiJERkE4OkUyQUZBOjE1RUNBMDM6MUJFNDA1Qzo2OTkxN0U5OCIsInZpc2l0b3JfaWQiOiIyODQwNjI1MzczNjM4NDg4NTYiLCJyZWdpb25fZWRnZSI6ImlhZCIsInJlZ2lvbl9yZW5kZXIiOiJpYWQifQ== |
| visitor-hmac | 2267e74a6287ef0739ef9eac5f1d62ccf1c37d6d9e038e6fb7eb3cee525c689f |
| hovercard-subject-tag | pull_request:1045806605 |
| github-keyboard-shortcuts | repository,pull-request-list,pull-request-conversation,pull-request-files-changed,copilot |
| google-site-verification | Apib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I |
| octolytics-url | https://collector.github.com/github/collect |
| analytics-location | / |
| fb:app_id | 1401488693436528 |
| apple-itunes-app | app-id=1477376905, app-argument=https://github.com/UnitTestBot/UTBotJava/pull/854/files |
| twitter:image | https://avatars.githubusercontent.com/u/336793?s=400&v=4 |
| twitter:card | summary_large_image |
| og:image | https://avatars.githubusercontent.com/u/336793?s=400&v=4 |
| og:image:alt | Description Suppose we have two executions with the same coverage, one produced by the symbolic engine, and one produced by the fuzzer. From the user's point of view, they are equivalent, and t... |
| og:site_name | GitHub |
| og:type | object |
| hostname | github.com |
| expected-hostname | github.com |
| None | 42c603b9d642c4a9065a51770f75e5e27132fef0e858607f5c9cb7e422831a7b |
| turbo-cache-control | no-preview |
| diff-view | unified |
| go-import | github.com/UnitTestBot/UTBotJava git https://github.com/UnitTestBot/UTBotJava.git |
| octolytics-dimension-user_id | 87413538 |
| octolytics-dimension-user_login | UnitTestBot |
| octolytics-dimension-repository_id | 480810501 |
| octolytics-dimension-repository_nwo | UnitTestBot/UTBotJava |
| octolytics-dimension-repository_public | true |
| octolytics-dimension-repository_is_fork | false |
| octolytics-dimension-repository_network_root_id | 480810501 |
| octolytics-dimension-repository_network_root_nwo | UnitTestBot/UTBotJava |
| turbo-body-classes | logged-out env-production page-responsive full-width |
| disable-turbo | true |
| browser-stats-url | https://api.github.com/_private/browser/stats |
| browser-errors-url | https://api.github.com/_private/browser/errors |
| release | 848bc6032dcc93a9a7301dcc3f379a72ba13b96e |
| ui-target | full |
| theme-color | #1e2327 |
| color-scheme | light dark |
Links:
Viewport: width=device-width