René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"⚙️ Extend the git-repo-scanner with an activity filter","articleBody":"\u003c!--\r\nThank you for reporting an issue in our documentation 🙌\r\n\r\nBefore opening a new issue, please make sure that we do not have any duplicates already open. You can ensure this by searching the issue list for this repository. If there is a duplicate, please close your issue and add a comment to the existing issue instead.\r\n--\u003e\r\n\r\n**Is your feature request related to a problem? Please describe.**\r\n\u003c!-- A clear and concise description of what the problem is. Ex. I'm always frustrated when [...] --\u003e\r\nAs secureCodeBox user i am heavily using the `git-repo-scanner` in combination with the `gitleaks` scanner via cascadingRules. When scanning large Gitlab oder GitHub organisations _with thousands of repos_ on a regular basis (daily, weekly) this leeds to thousands of independent `gitleaks` scans. \r\n\r\nTo save resources on the `gitleaks` side we extended `gitleaks` to be able to analyse all commits since a given timeframe (last 24h, last week,...): https://github.com/zricethezav/gitleaks/pull/498\r\n\r\nProblem is that the `git-repo-scanner` always returns all git repositories, even if they had no activity in the last timeframe. To safe even more resources it would be great to be able to configure an activity timeframe and use it as filter.\r\n\r\n**Describe the solution you'd like**\r\n\u003c!-- A clear and concise description of what you want to happen. --\u003e\r\nAdd the following configuration options to the `git-repo-scanner`:\r\n\r\n```\r\n--activity-since-duration= Return git repo findings with repo activity (e.g. commits) more recent than a specific date expresed by an duration (now + duration). A duration string is a possibly signed sequence of decimal numbers, each\r\n                               with optional fraction and a unit suffix, such as '300ms', '-1.5h' or '2h45m'. Valid time units are 'ns', 'us' (or 'µs'), 'ms', 's', 'm', 'h'.\r\n--activity-until-duration= Return git repo findings with repo activity (e.g. commits) older than a specific date expresed by an duration (now + duration). A duration string is a possibly signed sequence of decimal numbers, each with\r\n                               optional fraction and a unit suffix, such as '300ms', '-1.5h' or '2h45m'. Valid time units are 'ns', 'us' (or 'µs'), 'ms', 's', 'm', 'h'.\r\n```\r\n_Remark 🚧  : The concrete semantic of the timeframe definition can depend on the implementation details and might be slightly different. It is not important to implement it in exact that semantic._\r\n\r\n### Implementation Hints:\r\n- The GitLab Python client has some helpful methods like sorting all projects by `last_activity_at`: https://python-gitlab.readthedocs.io/en/stable/gl_objects/projects.html#examples \r\n  - `gl.projects.list(all=True, include_subgroups=True, order_by='last_activity_at', sort='desc')`\r\n  - could be used at: https://github.com/secureCodeBox/secureCodeBox/blob/fab01bdff212dde428e4dc9eb6eae4f34e211a98/scanners/git-repo-scanner/scanner/git_repo_scanner.py#L131\r\n- The Github Python client has some helpful methods like sorting all projects by last update https://pygithub.readthedocs.io/en/latest/github_objects/Organization.html#github.Organization.Organization.get_repos\r\n  - `get_repos(type='all', sort='updated', direction='desc')`\r\n  - could be used at: https://github.com/secureCodeBox/secureCodeBox/blob/fab01bdff212dde428e4dc9eb6eae4f34e211a98/scanners/git-repo-scanner/scanner/git_repo_scanner.py#L181\r\n\r\n### Example ScheduledScan Configuration\r\nThe following scheduled scan example will start a new `git-repo-scanner` scan every 24h and then cascades all the results with follow up `gitleaks` scans. The `gitleaks`scanner then analyses all the commits done within the last 24h timeframe.\r\n\r\n```yaml\r\napiVersion: \"execution.securecodebox.io/v1\"\r\nkind: ScheduledScan\r\nmetadata:\r\n  name: \"scb-github-repos\"\r\n  labels:\r\n    product: \"secureCodeBox\"\r\nspec:\r\n  interval: 24h\r\n  successfulJobsHistoryLimit: 1\r\n  failedJobsHistoryLimit: 1\r\n  scanSpec:\r\n    scanType: \"git-repo-scanner\"\r\n    parameters:\r\n      # configuration of the target git system\r\n      - \"--git-type\"\r\n      - \"github\"\r\n      - \"--organization\"\r\n      - \"secureCodeBox\"\r\n      # Provide an access token from ENV Vars defined via secret\r\n      - \"--access-token\"\r\n      - \"$(GITHUB_TOKEN)\"\r\n      # Filter findings - only return git repos with activity (commits) within a given timeframe \r\n      - \"--activity-since-duration\"\r\n      - \"24h\"\r\n    env:\r\n      - name: GITHUB_TOKEN\r\n        valueFrom:\r\n          secretKeyRef:\r\n            name: github-access-token\r\n            key: token\r\n    cascades:\r\n      matchLabels:\r\n        securecodebox.io/intensive: medium\r\n        securecodebox.io/invasive: non-invasive\r\n```\r\n\r\n```yaml\r\napiVersion: \"cascading.securecodebox.io/v1\"\r\nkind: CascadingRule\r\nmetadata:\r\n  name: \"gitleaks-github-scan-public\"\r\n  labels:\r\n    securecodebox.io/invasive: non-invasive\r\n    securecodebox.io/intensive: medium\r\nspec:\r\n  matches:\r\n    anyOf:\r\n      - name: \"GitHub Repo\"\r\n        attributes:\r\n          visibility: public\r\n  scanSpec:\r\n    scanType: \"gitleaks\"\r\n    parameters:\r\n      - \"--repo-url\"\r\n      - \"{{{attributes.web_url}}}\"\r\n      # Apply all available rules\r\n      - \"--config-path\"\r\n      - \"/home/config_all.toml\"\r\n      # Redact secrets from log messages and leaks\r\n      - \"--redact\"\r\n      # Only scan commits since the last 24h\r\n      - \"--commit-since-duration\"\r\n      - \"24h\"\r\n      # Provide an access token from ENV Vars defined via secret\r\n      - \"--access-token\"\r\n      - \"$(GITHUB_TOKEN)\"\r\n    env:\r\n      - name: GITHUB_TOKEN\r\n        valueFrom:\r\n          secretKeyRef:\r\n            name: github-access-token\r\n            key: token\r\n```\r\n\r\n**Describe alternatives you've considered**\r\n\u003c!-- A clear and concise description of any alternative solutions or features you've considered. --\u003e\r\nnone\r\n\r\n**Additional context**\r\n\u003c!-- Add any other context or screenshots about the feature request here. --\u003e\r\nnone","author":{"url":"https://github.com/rfelber","@type":"Person","name":"rfelber"},"datePublished":"2021-03-09T10:43:08.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":2},"url":"https://github.com/320/secureCodeBox/issues/320"}