René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"DynamoDB tags may be removed after repeated store.apply() due to asynchronous UntagResource / TagResource operations","articleBody":"## Expected Behavior \nWhen store.apply() is executed multiple times against a Feast registry configured with DynamoDB as the online store, table tags should remain stable and idempotent.\n\nRepeated executions of:\n\n`store.apply([..])`\n\nshould not modify the final tag state when no configuration changes have occurred.\n\nThe DynamoDB tables should consistently retain the configured tags after every execution.\n## Current Behavior\n\nFeast's DynamoDB online store implementation updates tags by:\n\n1. Reading existing tags (ListTagsOfResource)\n2. Removing all existing tags (UntagResource)\n3. Recreating all configured tags (TagResource)\n\nHowever, DynamoDB documents both TagResource and UntagResource as asynchronous and eventually consistent operations.\n\nBecause Feast invokes TagResource immediately after UntagResource without waiting for propagation, repeated executions can leave tables without tags.\n\nObserved behavior:\n\n```\napply #1 -\u003e tags present\napply #2 -\u003e tags missing\napply #3 -\u003e tags present\napply #4 -\u003e tags missing\n```\n\nAdding a delay between `UntagResource` and `TagResource` eliminates the issue.\n\n## Steps to reproduce\n\n### Feast configuration\n\nConfigure DynamoDB as the online store with table tags enabled:\n\n```\nonline_store:\n  type: dynamodb\n  region: eu-west-1\n\n  tags:\n    project: test-fstore\n```\n### Apply definitions repeatedly\n```\nstore.apply([...])\n```\n### Observe tag state\n\nRun repeatedly:\n\n```\naws dynamodb list-tags-of-resource \\\n  --resource-arn \u003ctable-arn\u003e\n\n```\nTags may disappear after an apply execution even though Feast issued a successful TagResource request.\n\n### Additional debugging\n\nInstrumenting `_update_tags()` shows:\n\n```\ncurrent_tags = dynamodb_client.list_tags_of_resource(...)\n\ndynamodb_client.untag_resource(...)\n\n# immediately after untag\nlist_tags_of_resource() -\u003e still returns previous tags\n\ntime.sleep(10)\n\nlist_tags_of_resource() -\u003e returns []\n\ndynamodb_client.tag_resource(...)\n\n# immediately after tag\nlist_tags_of_resource() -\u003e still returns []\n\ntime.sleep(10)\n\nlist_tags_of_resource() -\u003e returns expected tags\n```\n\nThis matches DynamoDB documentation stating that tag operations are eventually consistent.\n\n## Specifications\n\n- Version: Feast 0.63.0\n- Online Store: DynamoDB\n\n## Possible Solution\n\nThe DynamoDB online store implementation should not assume that `UntagResource` has completed when the API call returns.\nPotential fixes:\nOption 1\nWait until tag removal is visible before issuing `TagResource`.\nExample:\n```\nuntag_resource(...)\nwhile list_tags_of_resource(...):\n    time.sleep(1)\ntag_resource(...)\n```\nOption 2\nAvoid deleting and recreating all tags.\nInstead:\n\n\n- Compute the tag diff\n- Add/update only changed tags\n- Remove only obsolete tags\n\n\nThis would avoid the race condition entirely and reduce API calls.\nRelevant AWS documentation\n\nAWS DynamoDB documentation for UntagResource:\n\n\n\u003e UntagResource is an asynchronous operation.\n\n\n\u003e The application or removal of tags using TagResource and UntagResource APIs is eventually consistent.\n\n\n\u003e ListTagsOfResource API will only reflect the changes after a few seconds.\n\nThis behavior appears incompatible with the current implementation that performs `UntagResource` immediately followed by `TagResource` without waiting for propagation.","author":{"url":"https://github.com/parodin","@type":"Person","name":"parodin"},"datePublished":"2026-05-19T10:44:05.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":3},"url":"https://github.com/6418/feast/issues/6418"}