René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"DefectDojo persistence provider error handling","articleBody":"## 🐞 Bug report\r\n\r\n### Describe the bug\r\nRunning a large-scale Amass scan, we noticed that the DefectDojo persistence provider was failing. With closer inspection, we found that DefectDojo crashes on importing some endpoints which it regards as invalid. These domain names generally include underscores.\r\n\r\nThe persistence provider then restarts and tries again (failing), and again, etc.. Each time it restarts, it creates a new duplicate DefectDojo test object.\r\n\r\nOn another note, since amass is officially supported by SCB, but not by the DefectDojo persistence provider (uses generic), is this issue something that SCB actively supports?\r\n\r\n### Steps To Reproduce\r\nRun the DefectDojo persistence provider and import list of findings with an invalid domain name.\r\n\r\n### Expected behavior\r\n1. The persistence provider should not crash.\r\n2. The persistence provider should be as atomic as possible (preventing duplicate imports).\r\n3. Discussion: how should DefectDojo and the persistence provider handle an improperly formatted generic findings?\r\n\r\n### System (please complete the following information):\r\n- secureCodeBox Version: 3.0.1\r\n- DefectDojo version 2.1.0\r\n\r\n### Screenshots / Logs\r\n```\r\n[30/Aug/2021 20:17:01] ERROR [django.request:224] Internal Server Error: /api/v2/reimport-scan/\r\nTraceback (most recent call last):\r\n  File \"/app/./dojo/importers/reimporter/reimporter.py\", line 162, in process_parsed_findings\r\n    endpoint.clean()\r\n  File \"/app/./dojo/models.py\", line 1218, in clean\r\n    raise ValidationError(errors)\r\ndjango.core.exceptions.ValidationError: ['Host \"_invalid._host.com\" has invalid format']\r\n[...]\r\n```\r\n```\r\n2021-08-30 20:14:21 INFO  VersionedEngagementsStrategy:191 - Using ProductType Id: 1\r\nException in thread \"main\" org.springframework.web.client.HttpServerErrorException$InternalServerError: 500 Internal Server Error: [\r\n\u003c!doctype html\u003e\r\n\u003chtml lang=\"en\"\u003e\r\n\u003chead\u003e\r\n  \u003ctitle\u003eServer Error (500)\u003c/title\u003e\r\n\u003c/head\u003e\r\n\u003cbody\u003e\r\n  \u003ch1\u003eServer Error (500)\u003c/h1\u003e\u003cp\u003e\u003c/p\u003e\r\n\u003c/body\u003e\r\n\u003c/html\u003e\r\n]\r\n\tat org.springframework.web.client.HttpServerErrorException.create(HttpServerErrorException.java:100)\r\n\tat org.springframework.web.client.DefaultResponseErrorHandler.handleError(DefaultResponseErrorHandler.java:188)\r\n\tat org.springframework.web.client.DefaultResponseErrorHandler.handleError(DefaultResponseErrorHandler.java:125)\r\n\tat org.springframework.web.client.ResponseErrorHandler.handleError(ResponseErrorHandler.java:63)\r\n\tat org.springframework.web.client.RestTemplate.handleResponse(RestTemplate.java:819)\r\n\tat org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:777)\r\n\tat org.springframework.web.client.RestTemplate.execute(RestTemplate.java:711)\r\n\tat org.springframework.web.client.RestTemplate.exchange(RestTemplate.java:602)\r\n\tat io.securecodebox.persistence.defectdojo.service.ImportScanService.createFindings(ImportScanService.java:100)\r\n\tat io.securecodebox.persistence.defectdojo.service.ImportScanService.reimportScan(ImportScanService.java:117)\r\n\tat io.securecodebox.persistence.strategies.VersionedEngagementsStrategy.run(VersionedEngagementsStrategy.java:90)\r\n\tat io.securecodebox.persistence.DefectDojoPersistenceProvider.main(DefectDojoPersistenceProvider.java:42)\r\n```\r\n","author":{"url":"https://github.com/EndPositive","@type":"Person","name":"EndPositive"},"datePublished":"2021-08-31T07:56:43.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":10},"url":"https://github.com/618/secureCodeBox/issues/618"}