Title: runBenchmark setup needs to be awaited · Issue #36986 · angular/angular · GitHub
Open Graph Title: runBenchmark setup needs to be awaited · Issue #36986 · angular/angular
X Title: runBenchmark setup needs to be awaited · Issue #36986 · angular/angular
Description: 🐞 bug report Is this a regression? No Context perf_util.ts defines the function runBenchmark which is a wrapper for benchpress's sampler. The function does three key things: 1. Open the browser to a given url, 2. Allow for some initial s...
Open Graph Description: 🐞 bug report Is this a regression? No Context perf_util.ts defines the function runBenchmark which is a wrapper for benchpress's sampler. The function does three key things: 1. Open the browser to ...
X Description: 🐞 bug report Is this a regression? No Context perf_util.ts defines the function runBenchmark which is a wrapper for benchpress's sampler. The function does three key things: 1. Open the browser...
Opengraph URL: https://github.com/angular/angular/issues/36986
X: @github
Domain: github.com
{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"runBenchmark setup needs to be awaited","articleBody":"# 🐞 bug report\r\n\r\n\r\n### Is this a regression?\r\n\r\nNo\r\n\r\n\r\n### Context\r\n\r\n[perf_util.ts](https://github.com/angular/angular/blob/master/modules/e2e_util/perf_util.ts) defines the function `runBenchmark` which is a wrapper for benchpress's sampler. The function does three key things: 1. Open the browser to a given url, 2. Allow for some initial setup via a callback method \"`setup`\", and 3. Run a benchpress sampler to run `prepare` and `work` over and over again to get performance metrics.\r\n\r\nThis bug is specifically surrounding the `setup` callback, step 2 as described above.\r\n\r\n\r\n### Description\r\n\r\nBecause we do not await `config.setup()` in [perf_util.ts L37](https://github.com/angular/angular/blob/master/modules/e2e_util/perf_util.ts#L37), if setup is an async function (which is very likely since setup code will probably be interacting with the dom) there is no guarantee the call to `config.setup()` will finish before the sampling begins.\r\n\r\n\r\n## 🔬 Minimal Reproduction\r\nHere is a simple test I wrote\r\n\r\n```ts\r\n it('click (unchecked -> checked)', async() => {\r\n await runBenchmark({\r\n id: 'checkbox-overview-click-unchecked-to-checked',\r\n url: '',\r\n ignoreBrowserSynchronization: true,\r\n params: [],\r\n setup: async () => {\r\n await $('#show').click();\r\n console.log('setup before');\r\n await $('mat-checkbox').click();\r\n console.log('setup after');\r\n },\r\n prepare: async () => {\r\n console.log('prepare before');\r\n await $('mat-checkbox').click();\r\n console.log('prepare after');\r\n },\r\n work: async () => {\r\n console.log('execute before');\r\n await $('mat-checkbox').click();\r\n console.log('execute after');\r\n },\r\n });\r\n });\r\n```\r\n\r\nAnd here are the logs from that test\r\n```\r\nRunning protractor tests\r\n-----------------------------------------------------------------------------\r\nversion : 5.4.3\r\npwd : /private/var/tmp/_bazel_wagnermaciel/7b9bf2897304b51bc8634c41fff3e07f/execroot/angular_material/bazel-out/darwin-fastbuild/bin/test/benchmarks/material/checkbox/benchmark_chromium.sh.runfiles/angular_material\r\nconf : /private/var/tmp/_bazel_wagnermaciel/7b9bf2897304b51bc8634c41fff3e07f/execroot/angular_material/bazel-out/darwin-fastbuild/bin/test/benchmarks/material/checkbox/benchmark_chromium.sh.runfiles/angular_material/test/benchmarks/material/checkbox/benchmark_wrapped_test.conf.js\r\nnode_options:\r\n\r\nYour protractor configuration specifies capabilities for browser 'chrome'\r\nwhich will be merged with capabilities provided by Bazel resulting in: {\r\n \"browserName\": \"chrome\",\r\n \"chromeOptions\": {\r\n \"args\": [\r\n \"--js-flags=--expose-gc\",\r\n \"--no-sandbox\",\r\n \"--headless\",\r\n \"--disable-dev-shm-usage\",\r\n \"--no-sandbox\",\r\n \"--headless\",\r\n \"--disable-gpu\",\r\n \"--disable-dev-shm-usage\"\r\n ],\r\n \"perfLoggingPrefs\": {\r\n \"traceCategories\": \"v8,blink.console,devtools.timeline,disabled-by-default-devtools.timeline,blink.user_timing\"\r\n },\r\n \"binary\": \"/private/var/tmp/_bazel_wagnermaciel/7b9bf2897304b51bc8634c41fff3e07f/execroot/angular_material/bazel-out/darwin-fastbuild/bin/test/benchmarks/material/checkbox/benchmark_chromium.sh.runfiles/npm_dev_infra/benchmark/browsers/chromium_archive.out/chrome-mac/Chromium.app/Contents/MacOS/chromium\"\r\n },\r\n \"loggingPrefs\": {\r\n \"performance\": \"ALL\",\r\n \"browser\": \"ALL\"\r\n }\r\n}\r\n[19:36:23] I/launcher - Running 1 instances of WebDriver\r\n[19:36:23] I/direct - Using ChromeDriver directly...\r\nServer listening on http://wagnermaciel-macbookpro.roam.corp.google.com:36947/\r\nStarted\r\n\r\n\r\n(node:30861) [DEP0005] DeprecationWarning: Buffer() is deprecated due to security and usability issues. Please use the Buffer.alloc(), Buffer.allocUnsafe(), or Buffer.from() methods instead.\r\n*\r\nsetup before\r\nBENCHMARK checkbox-overview-click-unchecked-to-checked\r\nDescription:\r\n- forceGc: false\r\n- regressionSlopeMetric: scriptTime\r\n- runId: 09423a40-909a-11ea-b84b-a70081dde6c8\r\n- sampleSize: 20\r\n- userAgent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_4) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/76.0.3809.0 Safari/537.36\r\nMetrics:\r\n- gcAmount: gc amount in kbytes\r\n- gcTime: gc time in ms\r\n- majorGcTime: time of major gcs in ms\r\n- pureScriptTime: script execution time in ms, without gc nor render\r\n- renderTime: render time in ms\r\n- scriptTime: script execution time in ms, including gc and render\r\n\r\n gcAmount | gcTime | majorGcTime | pureScriptTime | renderTime | scriptTime\r\n------------------ | ------------------ | ------------------ | ------------------ | ------------------ | ------------------\r\nprepare before\r\nsetup after\r\nprepare after\r\nexecute before\r\nexecute after\r\n 0.00 | 0.00 | 0.00 | 1.57 | 2.87 | 1.83\r\n```\r\n","author":{"url":"https://github.com/wagnermaciel","@type":"Person","name":"wagnermaciel"},"datePublished":"2020-05-07T19:46:35Z","url":"https://github.com/angular/angular/issues/36986"}
| react-profiling | 0 |
| react-import-map | react,react-dom,react-dom/client,react-dom/profiling,react-is,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@b8488ee2c322 |
| route-pattern | /_view_fragments/issues/show/:user_id/:repository/:id/issue_layout(.:format) |
| route-controller | voltron_issues_fragments |
| route-action | issue_layout |
| fetch-nonce | v2:44a36d83-c2cc-343a-9ad8-25ca7186c7f2 |
| current-catalog-service-hash | 81bb79d38c15960b92d99bca9288a9108c7a47b18f2423d0f6438c5b7bcd2114 |
| request-id | CE88:25598F:188ECDC:2068CEB:6B0140F4 |
| html-safe-nonce | 9ccf2cf5c9116fc845616c65611a342426f91d2bbcfd11eccb9a9080b7816110 |
| visitor-payload | eyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiJDRTg4OjI1NTk4RjoxODhFQ0RDOjIwNjhDRUI6NkIwMTQwRjQiLCJ2aXNpdG9yX2lkIjoiNjg2MjczOTA5MzcxMzkyODQzNiIsInJlZ2lvbl9lZGdlIjoiaWFkIiwicmVnaW9uX3JlbmRlciI6ImlhZCJ9 |
| visitor-hmac | 6ff7ba266a4d51e70a81ccf24451802e929436643f3e57f8829e697d76d40a1a |
| hovercard-subject-tag | issue:614301066 |
| github-keyboard-shortcuts | repository,issues,copilot |
| google-site-verification | Apib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I |
| octolytics-url | https://collector.github.com/github/collect |
| analytics-location | / |
| fb:app_id | 1401488693436528 |
| apple-itunes-app | app-id=1477376905, app-argument=https://github.com/_view_fragments/issues/show/angular/angular/36986/issue_layout |
| twitter:image | https://opengraph.githubassets.com/c90a596c2f04ff0bd83076def12dd489bc74eac0b1c0587fcc6e54eb0909dfdb/angular/angular/issues/36986 |
| twitter:card | summary_large_image |
| og:image | https://opengraph.githubassets.com/c90a596c2f04ff0bd83076def12dd489bc74eac0b1c0587fcc6e54eb0909dfdb/angular/angular/issues/36986 |
| og:image:alt | 🐞 bug report Is this a regression? No Context perf_util.ts defines the function runBenchmark which is a wrapper for benchpress's sampler. The function does three key things: 1. Open the browser to ... |
| og:image:width | 1200 |
| og:image:height | 600 |
| og:site_name | GitHub |
| og:type | object |
| og:author:username | wagnermaciel |
| hostname | github.com |
| expected-hostname | github.com |
| None | cc5dfdd32f62ebd3223cab3af3aa2b268d3605218237326a48a99d1efd89495c |
| turbo-cache-control | no-preview |
| go-import | github.com/angular/angular git https://github.com/angular/angular.git |
| octolytics-dimension-user_id | 139426 |
| octolytics-dimension-user_login | angular |
| octolytics-dimension-repository_id | 24195339 |
| octolytics-dimension-repository_nwo | angular/angular |
| octolytics-dimension-repository_public | true |
| octolytics-dimension-repository_is_fork | false |
| octolytics-dimension-repository_network_root_id | 24195339 |
| octolytics-dimension-repository_network_root_nwo | angular/angular |
| turbo-body-classes | logged-out env-production page-responsive |
| disable-turbo | false |
| browser-stats-url | https://api.github.com/_private/browser/stats |
| browser-errors-url | https://api.github.com/_private/browser/errors |
| release | 8c77c9e8439031ad54c75a250694842892f693e1 |
| ui-target | full |
| theme-color | #1e2327 |
| color-scheme | light dark |
| react-app-name | issues |
Links:
Viewport: width=device-width