René's URL Explorer Experiment


Title: JS: Monorepo support: bugfixes and separate from dependency installation by asgerf · Pull Request #3731 · github/codeql · GitHub

Open Graph Title: JS: Monorepo support: bugfixes and separate from dependency installation by asgerf · Pull Request #3731 · github/codeql

X Title: JS: Monorepo support: bugfixes and separate from dependency installation by asgerf · Pull Request #3731 · github/codeql

Description: This PR largely does four things: Make extraction more deterministic by sorting files and choosing more carefully to extract TypeScript files with the right tsconfig.json file. Bugfixes: Use LGTM_SRC as source root instead of CWD. The latter only worked in codeql. Inititialize fields correctly 🤦 Separate monorepo support from dependency installation. This means monorepo support is on by default but dependency installation is still feature flagged. To avoid an OOM when extracting kibana, restrict properties extracted for type alias types. Some projects do a lot of type meta-programming, where a type alias is essentially a function call at the type-level. From that point of view, the restriction on aliases makes sense IMO, and isn't as arbitrary as it might seem at first. I'm getting less and less sure about the value we're getting out of the incomplete type property graph we're extracting. It's something we might consider removing if it gets in the way of more important things. Evaluations Metrics on typescript-full shows overall more call edges and more typed expressions. The few regressions are due to sorting files in a way that happens to be worse for those snapshots. security/smoke-test was completely uneventful. security/nightly shows a new result from the new call edges. Based on the above evaluation I'd say the timings are just noise. A simple offline test shows the extraction time for kibana going from 2148s to 1808s, a speedup likely resulting from the restricted handling of aliasing. This JS-Differences job shows the extraction time for vscode increase by 2%. Dist upgrade After review, I'd like to do a distribution upgrade to verify that we don't break the world. Commit-by-commit review recommended.

Open Graph Description: This PR largely does four things: Make extraction more deterministic by sorting files and choosing more carefully to extract TypeScript files with the right tsconfig.json file. Bugfixes: Use LGTM...

X Description: This PR largely does four things: Make extraction more deterministic by sorting files and choosing more carefully to extract TypeScript files with the right tsconfig.json file. Bugfixes: Use LGTM...

Opengraph URL: https://github.com/github/codeql/pull/3731

X: @github

direct link

Domain: github.com

react-profiling0
react-import-mapreact,react-dom,react-dom/client,react-dom/profiling,react-is,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@b8488ee2c322
route-pattern/:user_id/:repository/pull/:id/files(.:format)
route-controllerpull_requests
route-actionfiles
fetch-noncev2:eabd2e7e-17b1-fc8d-263e-d9a227204777
current-catalog-service-hashae870bc5e265a340912cde392f23dad3671a0a881730ffdadd82f2f57d81641b
request-idBBD8:3D97C8:95E0B4:C9D3A3:6B00E806
html-safe-noncecef7732ec061fa86ff190fff7c002b25c43c2d946b1dd2377a7b7e94eff08135
visitor-payloadeyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiJCQkQ4OjNEOTdDODo5NUUwQjQ6QzlEM0EzOjZCMDBFODA2IiwidmlzaXRvcl9pZCI6IjE5ODkwODA5MzUxODE2NDE3MzQiLCJyZWdpb25fZWRnZSI6ImlhZCIsInJlZ2lvbl9yZW5kZXIiOiJpYWQifQ==
visitor-hmacab3a9e67cb86a05757fd5477e1614abd482d9d8d5ae2660f8c0f686a94b44cac
hovercard-subject-tagpull_request:435701754
github-keyboard-shortcutsrepository,pull-request-list,pull-request-conversation,pull-request-files-changed,copilot
google-site-verificationApib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I
octolytics-urlhttps://collector.github.com/github/collect
analytics-location///pull_requests/show/files
fb:app_id1401488693436528
apple-itunes-appapp-id=1477376905, app-argument=https://github.com/github/codeql/pull/3731/files
twitter:imagehttps://avatars.githubusercontent.com/u/316427?s=400&v=4
twitter:cardsummary_large_image
og:imagehttps://avatars.githubusercontent.com/u/316427?s=400&v=4
og:image:altThis PR largely does four things: Make extraction more deterministic by sorting files and choosing more carefully to extract TypeScript files with the right tsconfig.json file. Bugfixes: Use LGTM...
og:site_nameGitHub
og:typeobject
hostnamegithub.com
expected-hostnamegithub.com
None27088b6977484d0d3321d9a11c907c6d6637b315cfa359ef7952dbf6292f9b14
turbo-cache-controlno-preview
diff-viewunified
go-importgithub.com/github/codeql git https://github.com/github/codeql.git
octolytics-dimension-user_id9919
octolytics-dimension-user_logingithub
octolytics-dimension-repository_id143040428
octolytics-dimension-repository_nwogithub/codeql
octolytics-dimension-repository_publictrue
octolytics-dimension-repository_is_forkfalse
octolytics-dimension-repository_network_root_id143040428
octolytics-dimension-repository_network_root_nwogithub/codeql
turbo-body-classeslogged-out env-production page-responsive full-width
disable-turbotrue
browser-stats-urlhttps://api.github.com/_private/browser/stats
browser-errors-urlhttps://api.github.com/_private/browser/errors
release624179fb8b88d26ba5bba08894eb60a1c5a1d93d
ui-targetfull
theme-color#1e2327
color-schemelight dark

Links:

Skip to contenthttps://github.com/github/codeql/pull/3731/files#start-of-content
https://github.com/
Sign inhttps://github.com/login?return_to=https%3A%2F%2Fgithub.com%2Fgithub%2Fcodeql%2Fpull%2F3731%2Ffiles
GitHubRepository hosting, code review, and collaborationhttps://github.com/platform
CopilotAI coding agent for tasks, projects, and automated workhttps://github.com/features/copilot
ActionsCI/CD and automationhttps://github.com/features/actions
Code QualityQuality analysis, fixes, and enforcementhttps://github.com/features/code-quality
CodespacesCloud development environmentshttps://github.com/features/codespaces
Code SecurityVulnerability detection and remediationhttps://github.com/security/advanced-security/code-security
Secret ProtectionSecret scanning and leak preventionhttps://github.com/security/advanced-security/secret-protection
All Featureshttps://github.com/features
Documentationhttps://docs.github.com/en
Traininghttps://learn.github.com/skills
Certificationshttps://learn.github.com/credentials
Build with Copilothttps://github.com/features/ai
Customize Copilothttps://docs.github.com/en/copilot/how-tos/copilot-on-github/customize-copilot/customize-copilot-overview
Integrationshttps://github.com/marketplace
Trending reposhttps://github.com/trending
GitHub Sponsorshttps://github.com/open-source/sponsors
Developer articleshttps://github.com/resources/articles
ChangelogYour source for the latest features, improvements, and fixes across GitHub products, all in one place.https://github.blog/changelog/
Enterpriseshttps://github.com/enterprise
Teamhttps://github.com/team
Startupshttps://github.com/enterprise/startups
Nonprofitshttps://github.com/solutions/industry/nonprofits
Healthcarehttps://github.com/solutions/industry/healthcare
Financial serviceshttps://github.com/solutions/industry/financial-services
Manufacturinghttps://github.com/solutions/industry/manufacturing
Governmenthttps://github.com/solutions/industry/government
See all industrieshttps://github.com/solutions/industry
Why GitHubhttps://github.com/why-github
Customer Storieshttps://github.com/customer-stories
Business insightshttps://github.com/solutions/executive-insights
Trust Centerhttps://github.com/trust-center
GitHub Advanced SecurityEnterprise-grade code securityhttps://github.com/security/advanced-security
Copilot for businessEnterprise-grade AI featureshttps://github.com/features/copilot/copilot-business
Premium SupportEnterprise-grade 24/7 supporthttps://github.com/enterprise/premium-support
Documentationhttps://docs.github.com/en
Resource centerhttps://github.com/resources
Bloghttps://github.blog/
Events & webinarshttps://github.com/resources/events
Community Forumhttps://github.com/orgs/community/discussions
Customer Storieshttps://github.com/customer-stories
Integrationshttps://github.com/marketplace
Customer supporthttps://help.github.com/
Traininghttps://learn.github.com/skills
GitHub Statushttps://www.githubstatus.com/
Abouthttps://github.com/about
Partnershttps://github.com/partners
Changeloghttps://github.blog/changelog/
GitHub UniverseThe event for developers, by developers — talks, workshops, and what’s next for GitHub.https://githubuniverse.com/
Pricinghttps://github.com/pricing
Sign inhttps://github.com/login?return_to=https%3A%2F%2Fgithub.com%2Fgithub%2Fcodeql%2Fpull%2F3731%2Ffiles
Sign uphttps://github.com/signup?ref_cta=Sign+up&ref_loc=header+logged+out&ref_page=%2F%3Cuser-name%3E%2F%3Crepo-name%3E%2Fpull_requests%2Fshow%2Ffiles&source=header-repo&source_repo=github%2Fcodeql
Reloadhttps://github.com/github/codeql/pull/3731/files
Reloadhttps://github.com/github/codeql/pull/3731/files
Reloadhttps://github.com/github/codeql/pull/3731/files
githubhttps://github.com/github
codeqlhttps://github.com/github/codeql
Codehttps://github.com/github/codeql
Issues1k (1k)https://github.com/github/codeql/issues
Pull requests507 (507)https://github.com/github/codeql/pulls
Discussionshttps://github.com/github/codeql/discussions
Actionshttps://github.com/github/codeql/actions
Projectshttps://github.com/github/codeql/projects
Security and qualityhttps://github.com/github/codeql/security
Insightshttps://github.com/github/codeql/insights
Sign up for GitHub https://github.com/signup?return_to=%2Fgithub%2Fcodeql%2Fissues%2Fnew%2Fchoose
terms of servicehttps://docs.github.com/terms
privacy statementhttps://docs.github.com/privacy
Sign inhttps://github.com/login?return_to=%2Fgithub%2Fcodeql%2Fissues%2Fnew%2Fchoose
semmle-qlcihttps://github.com/semmle-qlci
github:masterhttps://github.com/github/codeql/tree/master
asger-semmle:js/monorepo-bugfixeshttps://github.com/asger-semmle/ql/tree/js/monorepo-bugfixes
Conversation 12 https://github.com/github/codeql/pull/3731
Commits 12 https://github.com/github/codeql/pull/3731/commits
Checks 0 https://github.com/github/codeql/pull/3731/checks
Findings https://github.com/github/codeql/pull/3731/findings
Files changed https://github.com/github/codeql/pull/3731/files
Please reload this pagehttps://github.com/github/codeql/pull/3731/files
JS: Monorepo support: bugfixes and separate from dependency installation https://github.com/github/codeql/pull/3731/files#top
Show all changes 12 commits https://github.com/github/codeql/pull/3731/files
cb0a249 JS: Sort files asgerf Jun 3, 2020 https://github.com/github/codeql/pull/3731/commits/cb0a2498b004857b5b56ce13af8836d4e8d31b23
aaf1417 JS: Fix source root asgerf Jun 3, 2020 https://github.com/github/codeql/pull/3731/commits/aaf141782f1a3ee0bafab025a8226bdc3c35870b
dceb211 JS: Pass source root to Node.js process asgerf Jun 3, 2020 https://github.com/github/codeql/pull/3731/commits/dceb2110210f89c8a1b1df4ef12aa49baaf028a9
ba5d6bb JS: Actually set fields asgerf Jun 3, 2020 https://github.com/github/codeql/pull/3731/commits/ba5d6bb2e9e70f9a76d4fb4b0ce750724c068e98
6d15397 JS: Ensure we never write outside the scratch dir asgerf Jun 3, 2020 https://github.com/github/codeql/pull/3731/commits/6d15397fdc7b4ee597a11666ebde56ea5a26ed2f
6ff8137 JS: Also sort files in legacy extractor asgerf Jun 5, 2020 https://github.com/github/codeql/pull/3731/commits/6ff81377d5b5e5fce112c5597156053caf5a10a7
cf78475 JS: Only extract included files with a given tsconfig asgerf Jun 5, 2020 https://github.com/github/codeql/pull/3731/commits/cf784757997035b07430bc1eb96068122abf2a81
cc3e62f JS: Move stack trace limit to top of file asgerf Jun 12, 2020 https://github.com/github/codeql/pull/3731/commits/cc3e62f5357459aade88219cbc8d413b8ab016fa
4c4acd5 JS: Factor out loading of tsconfig files asgerf Jun 12, 2020 https://github.com/github/codeql/pull/3731/commits/4c4acd50bdb05ad8badb41c06213aadd90359766
675c64d JS: Prefer extracting file with tsconfig that included it asgerf Jun 12, 2020 https://github.com/github/codeql/pull/3731/commits/675c64d9d4b32957c0c821c7f80d60e2bab7c3c8
ad48c4e JS: Always prepare package.json files asgerf Jun 1, 2020 https://github.com/github/codeql/pull/3731/commits/ad48c4e54d1b80c698f61f1274eea2e97c2b1593
e28284b JS: Fix javadoc asgerf Jun 17, 2020 https://github.com/github/codeql/pull/3731/commits/e28284bd01298a83ec4bf46a4dda73c224a9af05
Clear filters https://github.com/github/codeql/pull/3731/files
Please reload this pagehttps://github.com/github/codeql/pull/3731/files
Please reload this pagehttps://github.com/github/codeql/pull/3731/files
main.ts https://github.com/github/codeql/pull/3731/files#diff-ae9cb8c7ab2b8622a84b032f9529817adc5c06f123ad107e03e0cef73c9a52bb
virtual_source_root.ts https://github.com/github/codeql/pull/3731/files#diff-2b3f8b9bba728db1693241a3c2a6daedc3f477ceed68157c604ef66a21de3353
AutoBuild.java https://github.com/github/codeql/pull/3731/files#diff-dfec202bb4425c99bf844bc2e4248e78fe2bda1e9f4e04e03ae51c92917acff0
DependencyInstallationResult.java https://github.com/github/codeql/pull/3731/files#diff-5266bf16a477dc888a46c2773b219d20d41d87017d593492a6fcecdb7f162d66
Main.java https://github.com/github/codeql/pull/3731/files#diff-7d2f65585212078eaf8063085cea23a530dd479378804beb85874d20877aa97b
AutoBuildTests.java https://github.com/github/codeql/pull/3731/files#diff-30dd4d715de0206169faef4ff210cc09960f2da8196c466bb45a4f54be79e844
ParsedProject.java https://github.com/github/codeql/pull/3731/files#diff-b982cf581685c770da9aa95cc7ca6ad052c56f0a9f4774a18fc3a25e3699f5e9
TypeScriptParser.java https://github.com/github/codeql/pull/3731/files#diff-75a9794607a7722a62e05e7c50e581e008e19a1b1e6cdc0bb3f736c436698de4
javascript/extractor/lib/typescript/src/main.tshttps://github.com/github/codeql/pull/3731/files#diff-ae9cb8c7ab2b8622a84b032f9529817adc5c06f123ad107e03e0cef73c9a52bb
View file https://github.com/asger-semmle/ql/blob/e28284bd01298a83ec4bf46a4dda73c224a9af05/javascript/extractor/lib/typescript/src/main.ts
Open in desktop https://desktop.github.com
https://github.co/hiddenchars
https://github.com/github/codeql/pull/3731/{{ revealButtonHref }}
https://github.com/github/codeql/pull/3731/files#diff-ae9cb8c7ab2b8622a84b032f9529817adc5c06f123ad107e03e0cef73c9a52bb
https://github.com/github/codeql/pull/3731/files#diff-ae9cb8c7ab2b8622a84b032f9529817adc5c06f123ad107e03e0cef73c9a52bb
https://github.com/github/codeql/pull/3731/files#diff-ae9cb8c7ab2b8622a84b032f9529817adc5c06f123ad107e03e0cef73c9a52bb
https://github.com/github/codeql/pull/3731/files#diff-ae9cb8c7ab2b8622a84b032f9529817adc5c06f123ad107e03e0cef73c9a52bb
https://github.com/github/codeql/pull/3731/files#diff-ae9cb8c7ab2b8622a84b032f9529817adc5c06f123ad107e03e0cef73c9a52bb
https://github.com/github/codeql/pull/3731/files#diff-ae9cb8c7ab2b8622a84b032f9529817adc5c06f123ad107e03e0cef73c9a52bb
https://github.com/github/codeql/pull/3731/files#diff-ae9cb8c7ab2b8622a84b032f9529817adc5c06f123ad107e03e0cef73c9a52bb
https://github.com/github/codeql/pull/3731/files#diff-ae9cb8c7ab2b8622a84b032f9529817adc5c06f123ad107e03e0cef73c9a52bb
https://github.com/github/codeql/pull/3731/files#diff-ae9cb8c7ab2b8622a84b032f9529817adc5c06f123ad107e03e0cef73c9a52bb
https://github.com/github/codeql/pull/3731/files#diff-ae9cb8c7ab2b8622a84b032f9529817adc5c06f123ad107e03e0cef73c9a52bb
https://github.com/github/codeql/pull/3731/files#diff-ae9cb8c7ab2b8622a84b032f9529817adc5c06f123ad107e03e0cef73c9a52bb
https://github.com/github/codeql/pull/3731/files#diff-ae9cb8c7ab2b8622a84b032f9529817adc5c06f123ad107e03e0cef73c9a52bb
https://github.com/github/codeql/pull/3731/files#diff-ae9cb8c7ab2b8622a84b032f9529817adc5c06f123ad107e03e0cef73c9a52bb
javascript/extractor/lib/typescript/src/virtual_source_root.tshttps://github.com/github/codeql/pull/3731/files#diff-2b3f8b9bba728db1693241a3c2a6daedc3f477ceed68157c604ef66a21de3353
View file https://github.com/asger-semmle/ql/blob/e28284bd01298a83ec4bf46a4dda73c224a9af05/javascript/extractor/lib/typescript/src/virtual_source_root.ts
Open in desktop https://desktop.github.com
https://github.co/hiddenchars
https://github.com/github/codeql/pull/3731/{{ revealButtonHref }}
https://github.com/github/codeql/pull/3731/files#diff-2b3f8b9bba728db1693241a3c2a6daedc3f477ceed68157c604ef66a21de3353
https://github.com/github/codeql/pull/3731/files#diff-2b3f8b9bba728db1693241a3c2a6daedc3f477ceed68157c604ef66a21de3353
Please reload this pagehttps://github.com/github/codeql/pull/3731/files
https://github.com
Termshttps://docs.github.com/site-policy/github-terms/github-terms-of-service
Privacyhttps://docs.github.com/site-policy/privacy-policies/github-privacy-statement
Securityhttps://github.com/security
Statushttps://www.githubstatus.com/
Communityhttps://github.community/
Docshttps://docs.github.com/
Contacthttps://support.github.com?tags=dotcom-footer

Viewport: width=device-width


URLs of crawlers that visited me.