René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Support for Custom S3 Services with a Custom Root CA","articleBody":"\u003c!--\r\nThank you for reporting an issue in our documentation 🙌\r\n\r\nBefore opening a new issue, please make sure that we do not have any duplicates already open. You can ensure this by searching the issue list for this repository. If there is a duplicate, please close your issue and add a comment to the existing issue instead.\r\n--\u003e\r\nAs secureCodeBox operations user i would like to integrate and configure custom S3 based persistence services instead of public available services like AWS Buckets, GCP Storage or DO. \r\n\r\n**Is your feature request related to a problem? Please describe.**\r\n\u003c!-- A clear and concise description of what the problem is. Ex. I'm always frustrated when [...] --\u003e\r\nBy now it is already possible to configure a custom S3 Backend as documented here: \r\n- https://docs.securecodebox.io/docs/getting-started/installation#using-a-hosted-s3-buckets-as-storage-backend\r\n- https://github.com/secureCodeBox/secureCodeBox/tree/main/operator#chart-configuration\r\n\r\nBut one feature is missing in the case of custom/internal S3 Services. It must be possible to add and configure custom root CA's for all secureCodeBox Components (Operator, Scanner, Hooks,...) to ensure a valid TLS based connection, if the custom S3 Service uses non public avaliable certificates.\r\n\r\n**Describe the solution you'd like**\r\n\u003c!-- A clear and concise description of what you want to happen. --\u003e\r\nImplement a configuration Option for all HelmCharts to add a custom Root CA based on a given configMap containing the CA Certificate.\r\n\r\nSCB Components involved here:\r\n\r\n- [x] SCB Operator: https://github.com/secureCodeBox/secureCodeBox/tree/main/operator\r\n- [x] SCB Lurcher: https://github.com/secureCodeBox/secureCodeBox/tree/main/lurcher\r\n- [x] All Scanner Parser ([parseDefinition](https://docs.securecodebox.io/docs/api/crds/parse-definition)):  https://github.com/secureCodeBox/secureCodeBox/tree/main/scanners ...\r\n- [x] All Hooks ([ScanCompletionHook](https://docs.securecodebox.io/docs/api/crds/scan-completion-hook)): https://github.com/secureCodeBox/secureCodeBox/tree/main/hooks ...\r\n\r\n## Implementation Hints:\r\nRegarding the parseDefinition and ScanCompletionHook CRDs we have already implemented the possibility for _ENV_, _Mount_ and _FileMount_ configurations in the corresponding SCB CRD and controller. Additional informations are documented in issue #212.\r\n\r\nWhat is missing here is to add this configuration option to all Scanner and Hooks HelmChart by default. The following hints will guide you through some necessary steps:\r\n\r\n### Hook Hints\r\n#### Existing CRD _ScanCompletionHook_ Definition\r\nhttps://github.com/secureCodeBox/secureCodeBox/blob/3f22b172db08914626e5f199b78d54c93609fccf/operator/apis/execution/v1/scancompletionhook.go#L38-L65\r\n\r\n#### Example Hook HelmChart Value Configuration:\r\nThe following code shows an example how the missing configuration option should look like (for all existing hook charts). Based on https://github.com/secureCodeBox/secureCodeBox/blob/main/hooks/declarative-subsequent-scans/values.yaml\r\n```yaml\r\n# example Hook configuration values\r\nhookJob:\r\n  # hookJob.ttlSecondsAfterFinished -- seconds after which the kubernetes job for the hook will be deleted. Requires the Kubernetes TTLAfterFinished controller: https://kubernetes.io/docs/concepts/workloads/controllers/ttlafterfinished/\r\n  ttlSecondsAfterFinished: null\r\n  # hookJob.env -- Optional environment variables mapped into each scanJob (see: https://kubernetes.io/docs/tasks/inject-data-application/define-environment-variable-container/)\r\n  env: []\r\n  # hookJob.envFrom -- Optional mount environment variables from configMaps or secrets (see: https://kubernetes.io/docs/tasks/inject-data-application/distribute-credentials-secure/#configure-all-key-value-pairs-in-a-secret-as-container-environment-variables)\r\n  envFrom: []\r\n  # hookJob.extraVolumes -- Optional Volumes mapped into each hookJob (see: https://kubernetes.io/docs/concepts/storage/volumes/)\r\n  extraVolumes:\r\n    - name: ca-workdir\r\n      emptyDir: {}\r\n  # hookJob.extraVolumeMounts -- Optional VolumeMounts mapped into each hookJob (see: https://kubernetes.io/docs/concepts/storage/volumes/)\r\n  extraVolumeMounts:\r\n    - mountPath: /etc/ssl/\r\n      name: ca-workdir\r\n```\r\n\r\n#### Example Hook HelmChart Template implementation:\r\nbased on: https://github.com/secureCodeBox/secureCodeBox/blob/main/hooks/declarative-subsequent-scans/templates/declerative-subsequent-scans-hook.yaml\r\n\r\n```yaml\r\napiVersion: \"execution.securecodebox.io/v1\"\r\nkind: ScanCompletionHook\r\nmetadata:\r\n  name: {{ include \"declarative-subsequent-scans.fullname\" . }}\r\nspec:\r\n  type: ReadOnly\r\n  image: \"{{ .Values.image.repository }}:{{ .Values.image.tag | default .Chart.Version }}\"\r\n  ttlSecondsAfterFinished: {{ .Values.hookJob.ttlSecondsAfterFinished }}\r\n  serviceAccountName: declarative-combined-scans\r\n  env:  {{ include \" .Values.hookJob.env\" . }}\r\n  volumes:  {{ include \" .Values.hookJob.extraVolumes\" . }}\r\n  volumeMounts:  {{ include \" .Values.hookJob.extraVolumeMounts\" . }}\r\n```\r\n\r\n### Parser Hints\r\n#### Existing CRD _parseType_ Definition\r\nhttps://github.com/secureCodeBox/secureCodeBox/blob/3f22b172db08914626e5f199b78d54c93609fccf/operator/apis/execution/v1/parsedefinition_types.go#L27-L49\r\n\r\n#### Example Scanner HelmChart Value Configuration:\r\nThe following code shows an example how the missing configuration option should look like (for all existing scanner charts). Based on: https://github.com/secureCodeBox/secureCodeBox/blob/main/scanners/amass/values.yaml\r\n\r\n```yaml\r\n# example Scanner (Parser) configuration values\r\nparseJob:\r\n  # parseJob.ttlSecondsAfterFinished -- seconds after which the kubernetes job for the hook will be deleted. Requires the Kubernetes TTLAfterFinished controller: https://kubernetes.io/docs/concepts/workloads/controllers/ttlafterfinished/\r\n  ttlSecondsAfterFinished: null\r\n  # parseJob.env -- Optional environment variables mapped into each scanJob (see: https://kubernetes.io/docs/tasks/inject-data-application/define-environment-variable-container/)\r\n  env: []\r\n  # parseJob.envFrom -- Optional mount environment variables from configMaps or secrets (see: https://kubernetes.io/docs/tasks/inject-data-application/distribute-credentials-secure/#configure-all-key-value-pairs-in-a-secret-as-container-environment-variables)\r\n  envFrom: []\r\n  # parseJob.extraVolumes -- Optional Volumes mapped into each hookJob (see: https://kubernetes.io/docs/concepts/storage/volumes/)\r\n  extraVolumes:\r\n    - name: ca-workdir\r\n      emptyDir: {}\r\n  # parseJob.extraVolumeMounts -- Optional VolumeMounts mapped into each hookJob (see: https://kubernetes.io/docs/concepts/storage/volumes/)\r\n  extraVolumeMounts:\r\n    - mountPath: /etc/ssl/\r\n      name: ca-workdir\r\n```\r\n#### Example Parser HelmChart Template implementation:\r\nbased on https://github.com/secureCodeBox/secureCodeBox/blob/main/scanners/amass/templates/amass-parse-definition.yaml:\r\n\r\n```yaml\r\napiVersion: \"execution.securecodebox.io/v1\"\r\nkind: ParseDefinition\r\nmetadata:\r\n  name: \"amass-jsonl\"\r\nspec:\r\n  handlesResultsType: amass-jsonl\r\n  image: \"{{ .Values.parserImage.repository }}:{{ .Values.parserImage.tag | default .Chart.Version }}\"\r\n  ttlSecondsAfterFinished: {{ .Values.parseJob.ttlSecondsAfterFinished }}\r\n  env:  {{ include \" .Values.parseJob.env\" . }}\r\n  volumes:  {{ include \" .Values.parseJob.extraVolumes\" . }}\r\n  volumeMounts:  {{ include \" .Values.parseJob.extraVolumeMounts\" . }}\r\n```\r\n\r\n### Operator and Lurcher Hints\r\nThe following code shows an example how the missing configuration option should look like for the operator and lurcher:\r\n\r\n```yaml\r\n# example Lurcher configuration values\r\nlurcher:\r\n  # lurcher.env -- Optional environment variables mapped into the lurcher (see: https://kubernetes.io/docs/tasks/inject-data-application/define-environment-variable-container/)\r\n  env: []\r\n  # lurcher.envFrom -- Optional mount environment variables from configMaps or secrets (see: https://kubernetes.io/docs/tasks/inject-data-application/distribute-credentials-secure/#configure-all-key-value-pairs-in-a-secret-as-container-environment-variables)\r\n  envFrom: []\r\n  # lurcher.extraVolumes -- Optional Volumes mapped into the lurcher (see: https://kubernetes.io/docs/concepts/storage/volumes/)\r\n  extraVolumes:\r\n    - name: ca-workdir\r\n      emptyDir: {}\r\n  # lurcher.extraVolumeMounts -- Optional VolumeMounts mapped into the lurcher (see: https://kubernetes.io/docs/concepts/storage/volumes/)\r\n  extraVolumeMounts:\r\n    - mountPath: /etc/ssl/\r\n      name: ca-workdir\r\n\r\n# example Operator configuration values\r\n# env -- Optional environment variables mapped into the operator (see: https://kubernetes.io/docs/tasks/inject-data-application/define-environment-variable-container/)\r\nenv: []\r\n# envFrom -- Optional mount environment variables from configMaps or secrets (see: https://kubernetes.io/docs/tasks/inject-data-application/distribute-credentials-secure/#configure-all-key-value-pairs-in-a-secret-as-container-environment-variables)\r\nenvFrom: []\r\n# volumes -- Optional Volumes mapped into the operator (see: https://kubernetes.io/docs/concepts/storage/volumes/)\r\nvolumes:\r\n  - name: ca-workdir\r\n    emptyDir: {}\r\n# volumeMounts -- Optional VolumeMounts mapped into the operator (see: https://kubernetes.io/docs/concepts/storage/volumes/)\r\nvolumeMounts:\r\n  - mountPath: /etc/ssl/\r\n    name: ca-workdir\r\n```\r\n\r\n**Describe alternatives you've considered**\r\n\u003c!-- A clear and concise description of any alternative solutions or features you've considered. --\u003e\r\nnone\r\n\r\n**Additional context**\r\n\u003c!-- Add any other context or screenshots about the feature request here. --\u003e\r\n","author":{"url":"https://github.com/rfelber","@type":"Person","name":"rfelber"},"datePublished":"2021-02-11T13:05:08.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":1},"url":"https://github.com/297/secureCodeBox/issues/297"}