René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Runner exits with failures while browser tests report none","articleBody":"### Bug\r\n\r\nFor unknown reasons, `browserstack-runner` sometimes exits with failures while all tests run without errors.\r\n\r\nFor exemple see the following Travis job with `mocha`: all tests passed (or are skipped) but `browserstack-runner` exists with `failures: 3`.\r\n```log\r\nUsing config: /home/travis/build/ncoden/foundation-sites/browserstack.json\r\nLaunching 18 worker(s) for 18 run(s).\r\n[OS X Sierra, Safari 10.1] Passed: 260 tests, 252 passed, 0 failed, 8 skipped; ran for 6546ms\r\n[Windows 10, Internet Explorer 11.0] Passed: 260 tests, 252 passed, 0 failed, 8 skipped; ran for 14034ms\r\n[Windows 8, Chrome 65.0] Passed: 260 tests, 252 passed, 0 failed, 8 skipped; ran for 6260ms\r\n[Windows 10, Chrome 66.0] Passed: 260 tests, 252 passed, 0 failed, 8 skipped; ran for 7288ms\r\n[Windows 7, Internet Explorer 10.0] Passed: 260 tests, 252 passed, 0 failed, 8 skipped; ran for 15411ms\r\n[Windows 10, Internet Explorer 11.0] Passed: 260 tests, 252 passed, 0 failed, 8 skipped; ran for 16303ms\r\n[Windows 8.1, Firefox 60.0] Passed: 260 tests, 252 passed, 0 failed, 8 skipped; ran for 10193ms\r\n[Windows 7, Opera 53.0] Passed: 260 tests, 252 passed, 0 failed, 8 skipped; ran for 5517ms\r\n[ios 7.0, Mobile Safari null] Passed: 260 tests, 252 passed, 0 failed, 8 skipped; ran for 18048ms\r\n[Windows 10, Edge 17.0] Passed: 260 tests, 252 passed, 0 failed, 8 skipped; ran for 9909ms\r\n[Windows 7, Opera 52.0] Passed: 260 tests, 252 passed, 0 failed, 8 skipped; ran for 5855ms\r\n[OS X High Sierra, Safari 11.1] Passed: 260 tests, 252 passed, 0 failed, 8 skipped; ran for 9277ms\r\n[Windows 10, Firefox 61.0 beta] Passed: 260 tests, 252 passed, 0 failed, 8 skipped; ran for 10925ms\r\n[Windows 10, Edge 16.0] Passed: 260 tests, 252 passed, 0 failed, 8 skipped; ran for 11188ms\r\n[android 8.0, Android Browser null] Passed: 260 tests, 252 passed, 0 failed, 8 skipped; ran for 7173ms\r\nAll tests done, failures: 3.\r\nExit with fail due to some tests failure.\r\n```\r\n\r\nI found in the following in trace logs, I don't know why `android_browser_4_4` is not recognized, but I don't think this is related.\r\n```\r\n[android 4.4, Android Browser null] client.createWorker | response: undefined Error: Validation Failed - `device` invalid\r\n    at IncomingMessage.\u003canonymous\u003e (/home/travis/build/ncoden/foundation-sites/node_modules/browserstack/lib/client.js:73:9)\r\n    at IncomingMessage.emit (events.js:187:15)\r\n    at endReadableNT (_stream_readable.js:1090:12)\r\n    at process._tickCallback (internal/process/next_tick.js:63:19)\r\nError from BrowserStack:  Error: Validation Failed - `device` invalid\r\n    at IncomingMessage.\u003canonymous\u003e (/home/travis/build/ncoden/foundation-sites/node_modules/browserstack/lib/client.js:73:9)\r\n    at IncomingMessage.emit (events.js:187:15)\r\n    at endReadableNT (_stream_readable.js:1090:12)\r\n    at process._tickCallback (internal/process/next_tick.js:63:19)\r\n```\r\n\r\nSee also:\r\n* Browserstack config file: https://github.com/zurb/foundation-sites/blob/develop/browserstack.json\r\n* Full log: https://travis-ci.org/ncoden/foundation-sites/jobs/387169582\r\n* Full log with trace log level: https://travis-ci.org/ncoden/foundation-sites/jobs/387194396\r\n\r\n### Possible solution\r\n\r\nI checked at the source code. I have no problem with Node, but the code around the worker execution and results processing is very confusing. I am unable to propose a solution for now.\r\n\r\n","author":{"url":"https://github.com/ncoden","@type":"Person","name":"ncoden"},"datePublished":"2018-06-02T19:46:50.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":2},"url":"https://github.com/201/browserstack-runner/issues/201"}