René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Elasticsearch 8.x compatibility issue: persistence-elastic hook fails with TypeError on bulk indexing","articleBody":"## Description\n\nThe `persistence-elastic` hook (version 5.2.0) fails when attempting to persist scan findings to Elasticsearch 8.x due to an API response format incompatibility. The hook works correctly with Elasticsearch 7.x but crashes with a `TypeError` when using Elasticsearch 8.5.1 or later.\n\n \n\n## Error Message\n\nTypeError: Cannot read properties of undefined (reading 'errors') at handle (file:///home/app/hook-wrapper/hook/hook.js:110:22)\n\n \n\n \n\n## Impact\n\n \n\n- **Severity:** Medium\n\n- **Affected Users:** users running Elasticsearch 8.x with persistence-elastic hook\n\n- **Functionality:** Complete persistence failure - findings are not indexed in Elasticsearch\n\n- **Workaround:** Downgrade to Elasticsearch 7.17.x (not ideal for production)\n\n \n\n## Steps to Reproduce\n\n \n\n   ```bash\n\n   helm install elasticsearch elastic/elasticsearch \\\n\n     --version 8.5.1 \\\n\n     --set replicas=1 \\\n\n     --namespace default\n\n \n\nhelm install persistence-elastic \\\n\n  oci://[ghcr.io/securecodebox/helm/persistence-elastic](http://ghcr.io/securecodebox/helm/persistence-elastic) \\\n\n  --version 5.2.0 \\\n\n  --namespace default\n\n \n\nkubectl apply -f - \u003c\u003cEOF\n\napiVersion: \"[execution.securecodebox.io/v1](http://execution.securecodebox.io/v1)\"\n\nkind: Scan\n\nmetadata:\n\n  name: trivy-test\n\nspec:\n\n  scanType: \"trivy-image\"\n\n  parameters:\n\n    - \"nginx:latest\"\n\nEOF\n\n \n\nkubectl logs -l [app.kubernetes.io/name=persistence-elastic](http://app.kubernetes.io/name=persistence-elastic)\n\n \n\n \n\nExpected Behavior\n\nThe persistence hook should successfully index findings to Elasticsearch 8.x, just as it does with Elasticsearch 7.x.\n\n \n\nActual Behavior\n\nThe persistence hook crashes with TypeError: Cannot read properties of undefined (reading 'errors') when attempting to process the Elasticsearch bulk API response.\n\n \n\nRoot Cause Analysis\n\nThe issue is in the hook's handling of the Elasticsearch bulk API response. The response format changed between Elasticsearch 7.x and 8.x:\n\n \n\nElasticsearch 7.x response:\n\n{\n\n  body: {\n\n    errors: false,\n\n    items: [...]\n\n  }\n\n}\n\n \n\nElasticsearch 8.x response:\n\n{\n\n  errors: false,\n\n  items: [...]\n\n  // No .body wrapper\n\n}\n\n \n\nCurrent hook code (approximately line 110 in hook.js):\n\nconst { body: bulkResponse } = await client.bulk({ refresh: true, body });\n\n \n\nif (bulkResponse.errors) {  // ← TypeError here when bulkResponse is undefined\n\n  console.error(\"Bulk Request had errors:\");\n\n  console.log(bulkResponse);\n\n}\n\n \n\n \n\nWhen using Elasticsearch 8.x, the destructuring { body: bulkResponse } results in bulkResponse being undefined because the response doesn't have a .body property. This causes the TypeError when trying to access bulkResponse.errors.\n\n \n\nProposed Solution\n\nImplement a backwards-compatible fix that works with both Elasticsearch 7.x and 8.x:\n\n \n\n// Proposed fix for hook.js around line 110\n\nconst response = await client.bulk({ refresh: true, body });\n\n \n\n// Support both ES 7.x (response.body) and ES 8.x (response directly)\n\nconst bulkResponse = response.body || response;\n\n \n\nif (bulkResponse.errors) {\n\n  console.error(\"Bulk Request had errors:\");\n\n  console.log(bulkResponse);\n\n}\n\n \n\nThis change:\n\n \n\n✅ Works with Elasticsearch 7.x (uses response.body)\n\n✅ Works with Elasticsearch 8.x (uses response directly)\n\n✅ No breaking changes to existing functionality\n\n✅ Simple one-line fix\n\n✅ Follows the pattern used by Elasticsearch's official Node.js client migration guide\n\nEnvironment\n\nSecureCodeBox Version: 4.9.0\n\npersistence-elastic Hook Version: 5.2.0\n\nElasticsearch Version: 8.5.1 (issue occurs), 7.17.3 (works correctly)\n\nKubernetes Version: 1.28\n\nInstallation Method: Helm\n\n \n\n \n\nWorkaround\n\nCurrently, the only workaround is to downgrade Elasticsearch to version 7.17.x:\n\nhelm install elasticsearch elastic/elasticsearch \\\n\n  --version 7.17.3 \\\n\n  --namespace default\n\n \n\n \n\nHowever, this prevents users from:\n\n \n\nUsing Elasticsearch 8.x features\n\nFollowing Elasticsearch's recommended upgrade path\n\nReceiving security updates for Elasticsearch 8.x\n\nAdditional Context\n\nAffected scanners: All scanners that produce findings (Trivy, Semgrep, Nmap, custom scanners)\n\n \n\nChunk size impact: The error occurs when processing chunks of findings. In our testing:\n\n \n\n0 findings: Hook succeeds (no bulk operation)\n\n1-50 findings: Hook crashes on first bulk operation\n\n50+ findings: Hook crashes on first chunk (50 findings per chunk)\n\n \n\nrelated Elasticsearch migration documentation:\n\n \n\n[Elasticsearch 8.0 Breaking Changes](vscode-file://vscode-app/c:/Users/GKY24/Downloads/VSCode-win32-x64-1.104.1/resources/app/out/vs/code/electron-browser/workbench/workbench.html)\n\n[Elasticsearch Node.js Client Migration](vscode-file://vscode-app/c:/Users/GKY24/Downloads/VSCode-win32-x64-1.104.1/resources/app/out/vs/code/electron-browser/workbench/workbench.html)\n\n \n\nRequest\n\nPlease consider implementing the proposed backwards-compatible fix in the next release of the persistence-elastic hook. This will enable users to use Elasticsearch 8.x while maintaining compatibility with 7.x installations.\n\n \n\nThank you for maintaining this excellent project! 🙏","author":{"url":"https://github.com/conleth","@type":"Person","name":"conleth"},"datePublished":"2025-11-21T23:13:07.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":2},"url":"https://github.com/3385/secureCodeBox/issues/3385"}