René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"🐞 The git_repo_scanner triggers an abuse detection mechanism at GitHub for large organisations ","articleBody":"\u003c!--\r\nThank you for reporting an issue in our project 🙌\r\n\r\nBefore opening a new issue, please make sure that we do not have any duplicates already open. You can ensure this by searching the issue list for this repository. If there is a duplicate, please close your issue and add a comment to the existing issue instead.\r\n--\u003e\r\n\r\n**Describe the bug**\r\n\u003c!-- A clear and concise description of what the bug is. --\u003e\r\n\r\nI am having an issue with a git-repo-scanner + gitleaks working in concert to scan my company's org, which has over 2k repos.  The scan is ending around 900 repos with the following error:\r\n\r\n```\r\nINFO:git_repo_scanner: 901 - crp-legacy-ui-dhtmlWins\r\nINFO:git_repo_scanner: Github API Exception: 403 -\u003e You have triggered an abuse detection mechanism. Please wait a few minutes before you try again.\r\n```\r\n\r\nThis is causing CrashLoopBackoff events with the pod.  Ultimately the job fails with error.\r\n\r\nI understand what the error is telling me.  My main question is:  What is the largest number of repos does this scanner support? I would like to scan my entire org in one run, but I have not been successful, with the number of repos and the GH API throttle limits.  The --ignore-repo is a start, but seems not too effective given my large list of repos.\r\n\r\nOther questions:\r\nIs there a way to throttle api calls? \r\nIs there a recommended number of batches I should run with?\r\nWould there be a way to pass an argument to break up the scans?\r\n\r\n**To Reproduce**\r\n\r\nSteps to reproduce the behavior:\r\n1. Install via helm:  see second screenshot below.\r\n2. Create a git-repo-scanner yaml file.\r\n3. Run the git-repo-scanner job on a github org with over 2000 repos and with cascadingrules triggering gitleaks.\r\n4. Tail the created job's log, for example: `k logs job/scan-gh-repo-scan-org-coretech-wdbns -c git-repo-scanner -f `\r\n4. See 403 error in the first screenshot below.\r\n\r\n\r\n\r\n**Expected behavior**\r\n\u003c!-- A clear and concise description of what you expected to happen. --\u003e\r\n\r\n\r\n**System (please complete the following information):**\r\n\r\n - secureCodeBox 2.4.0\r\n - OS: Linux\r\n - Kubernetes Version [command: `kubectl version`]\r\n \r\nClient Version: version.Info{Major:\"1\", Minor:\"20\", GitVersion:\"v1.20.4\", GitCommit:\"e87da0bd6e03ec3fea7933c4b5263d151aafd07c\", GitTreeState:\"clean\", BuildDate:\"2021-02-21T20:23:45Z\", GoVersion:\"go1.15.8\", Compiler:\"gc\", Platform:\"darwin/amd64\"}\r\nServer Version: version.Info{Major:\"1\", Minor:\"18+\", GitVersion:\"v1.18.9-eks-d1db3c\", GitCommit:\"d1db3c46e55f95d6a7d3e5578689371318f95ff9\", GitTreeState:\"clean\", BuildDate:\"2020-10-20T22:18:07Z\", GoVersion:\"go1.13.15\", Compiler:\"gc\", Platform:\"linux/amd64\"}\r\n - Docker Version [command: `docker -v`]\r\n \r\nDocker version 20.10.2, build 2291f61\r\n - Browser [e.g. chrome, safari, firefox,...]\r\n N/A\r\n\r\n\r\n**Screenshots / Logs**\r\n\u003c!-- If applicable, add screenshots to help explain your problem. --\u003e\r\n![image](https://user-images.githubusercontent.com/60161201/110996342-6c6b0d00-8341-11eb-89da-84604951f2f6.png)\r\n\r\n![image](https://user-images.githubusercontent.com/60161201/110996167-27df7180-8341-11eb-973e-a762bd9eecd3.png)\r\n\r\n**Additional context**\r\n\u003c!-- Add any other context about the problem here. --\u003e\r\n","author":{"url":"https://github.com/glermaidt","@type":"Person","name":"glermaidt"},"datePublished":"2021-03-12T20:48:05.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":8},"url":"https://github.com/325/secureCodeBox/issues/325"}