René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"[FEATURE] External Policy/Configuration Loading with `${include()}` Function","articleBody":"## Feature Description\nImplement a function-style syntax for loading policies, IAM roles, and other complex JSON/YAML configurations from external files in `stackql-deploy`, similar to Terraform's file loading capabilities.\n\n## Current Behavior\nCurrently, all policies, IAM role definitions, and other complex JSON configurations must be embedded directly within the `stackql-deploy` manifest file, making manifests verbose and difficult to maintain, especially for complex policy documents.\n\n## Desired Behavior\nAllow users to reference external JSON/YAML files for policies and complex configurations using a function-style syntax:\n\n```yaml\n- name: policy_document\n  value: \"${include('policies/s3_bucket_policy.json')}\"\n```\nThe `${include()}` function would:\n1. Load content from the specified JSON or YAML file\n2. Parse the file content into a native object structure\n3. Support variable substitution for template variables within the loaded file\n4. Replace the function call with the loaded and processed content\n\n## File Format Requirements\n- Support both JSON (.json) and YAML (.yaml, .yml) file formats\n- Automatically detect format based on file extension\n- Validate file syntax during loading\n- Support nested includes (files including other files, with cycle detection)\n\n## Use Cases\n- IAM role policies (like the cross_account_role in the manifest)\n- S3 bucket policies\n- Assume role policy documents\n- Storage configurations\n- Any complex nested JSON structure\n\n## Implementation Details\n1. Implement a new `include()` function in the stackql-deploy template processing engine (at the `resource.props` level)\n2. Add file path resolution relative to the manifest file location\n3. Support environment-specific overrides (e.g., `policy.dev.json` vs `policy.prod.json`)\n4. Process template variables inside the loaded files\n5. Implement error handling for missing or invalid files\n\n## Example\nConsider a complex IAM policy currently embedded in the manifest:\n\n**Current approach (policy embedded in manifest):**\n```yaml\n- name: aws/iam/cross_account_role\n  file: aws/iam/iam_role.iql\n  props:\n    - name: assume_role_policy_document\n      value:\n        Version: \"2012-10-17\"\n        Statement:\n          - Sid: \"\"\n            Effect: \"Allow\"\n            Principal:\n              AWS: \"arn:aws:iam::{{ databricks_aws_account_id }}:root\"\n            Action: \"sts:AssumeRole\"\n            Condition:\n              StringEquals:\n                sts:ExternalId: \"{{ databricks_account_id }}\"\n    - name: policies\n      value:\n        - PolicyDocument:\n            Statement:\n              - Sid: Stmt1403287045000\n                Effect: Allow\n                Action:\n                  - \"ec2:AllocateAddress\"\n                  - \"ec2:AssociateDhcpOptions\"\n                  # ... many more actions ...\n                Resource:\n                  - \"*\"\n            Version: '2012-10-17'\n          PolicyName: \"{{ stack_name }}-{{ stack_env }}-policy\"\n```\n\n**Proposed approach with external files:**\n```yaml\n- name: aws/iam/cross_account_role\n  file: aws/iam/iam_role.iql\n  props:\n    - name: assume_role_policy_document\n      value: \"${include('policies/assume_role_policy.json')}\"\n    - name: policies\n      value:\n        - PolicyDocument: \"${include('policies/ec2_policy.json')}\"\n          PolicyName: \"{{ stack_name }}-{{ stack_env }}-policy\"\n```\n\n**External file: `policies/assume_role_policy.json`:**\n```json\n{\n  \"Version\": \"2012-10-17\",\n  \"Statement\": [\n    {\n      \"Sid\": \"\",\n      \"Effect\": \"Allow\",\n      \"Principal\": {\n        \"AWS\": \"arn:aws:iam::{{ databricks_aws_account_id }}:root\"\n      },\n      \"Action\": \"sts:AssumeRole\",\n      \"Condition\": {\n        \"StringEquals\": {\n          \"sts:ExternalId\": \"{{ databricks_account_id }}\"\n        }\n      }\n    }\n  ]\n}\n```\n\n**External file: `policies/ec2_policy.json`:**\n```json\n{\n  \"Statement\": [\n    {\n      \"Sid\": \"Stmt1403287045000\",\n      \"Effect\": \"Allow\",\n      \"Action\": [\n        \"ec2:AllocateAddress\",\n        \"ec2:AssociateDhcpOptions\",\n        \"ec2:AssociateIamInstanceProfile\"\n        // ... more actions ...\n      ],\n      \"Resource\": [\"*\"]\n    }\n  ],\n  \"Version\": \"2012-10-17\"\n}\n```\n\n## Template Variable Processing\nTemplate variables (using the `{{ variable_name }}` syntax) in external files would be processed just like they are in the main manifest file, allowing for dynamic configuration:\n\n**Example with template variables in external file:**\n```json\n{\n  \"Version\": \"2012-10-17\",\n  \"Statement\": [\n    {\n      \"Effect\": \"Allow\",\n      \"Principal\": {\n        \"AWS\": \"arn:aws:iam::{{ databricks_aws_account_id }}:root\"\n      },\n      \"Action\": \"sts:AssumeRole\",\n      \"Condition\": {\n        \"StringEquals\": {\n          \"sts:ExternalId\": \"{{ databricks_account_id }}\"\n        }\n      }\n    }\n  ]\n}\n```\n## Benefits\n- Cleaner, more maintainable manifest files\n- Easier version control of individual policies\n- Ability to reuse common policies across multiple stacks\n- Better separation of concerns in the infrastructure code\n- Familiar function-style syntax similar to other IaC tools\n\n## Additional Considerations\n- Path resolution: Relative to manifest file or working directory?\n- Error handling for missing or invalid files\n- Support for partial includes (specific sections of a file)\n- Documentation for best practices in organizing policy files","author":{"url":"https://github.com/jeffreyaven","@type":"Person","name":"jeffreyaven"},"datePublished":"2025-07-22T00:41:05.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":0},"url":"https://github.com/57/stackql-deploy/issues/57"}