René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Apache CloudStack Usage Server repeatedly reprocesses historical usage from fixed start date and creates duplicate cloud_usage records","articleBody":"### problem\n\nWe identified a persistent duplication issue in the Apache CloudStack Usage Server related to usage_type=13 (Network Offering usage).\n\nThe Usage Server continuously reprocesses historical usage records starting from 2026-01-13 00:00:37, generating duplicate entries in the cloud_usage table every day.\n\nThe issue appears to affect Network Offering usage accounting specifically (usage_type=13).\n\nObserved behavior:\n\nDuplicate cloud_usage rows are inserted repeatedly for the same:\nvm_instance_id\noffering_id\nusage_type\nstart_date\nend_date\nraw_usage\nThe duplicate count increases daily.\nExample:\nrecords for 2026-01-14 currently have 112 duplicates\nrecords for 2026-01-15 currently have 111 duplicates\nrecords for 2026-01-16 currently have 110 duplicates\n\nThe pattern strongly suggests that the Usage Server is reprocessing the entire historical usage window every day instead of advancing the aggregation checkpoint.\n\nExample duplicate query:\n\nSELECT\n  vm_instance_id,\n  offering_id,\n  usage_type,\n  start_date,\n  end_date,\n  raw_usage,\n  COUNT(*) AS duplicates\nFROM cloud_usage\nWHERE usage_type = 13\nGROUP BY\n  vm_instance_id,\n  offering_id,\n  usage_type,\n  start_date,\n  end_date,\n  raw_usage\nHAVING COUNT(*) \u003e 1\nORDER BY duplicates DESC;\n\nExample result:\n\nvm_instance_id = 456\noffering_id    = 39\nusage_type     = 13\nstart_date     = 2026-01-14 00:00:00\nend_date       = 2026-01-14 23:59:59\nraw_usage      = 24\nduplicates     = 112\n\nWe also observed repeated failed usage_job entries with invalid processing windows:\n\nstart_date = 2026-05-06 00:00:00\nend_date   = 2026-05-05 23:59:59\nsuccess    = 0\n\nAt the same time, successful jobs continuously reprocess the same historical range:\n\nstart_date = 2026-01-13 00:00:37\nend_date   = 2026-05-05 23:59:59\n\nThis causes:\n\ncontinuous duplicate billing records\ninflated usage data\nincorrect accounting\ncorrupted billing exports/reports\n\n### versions\n\nApache CloudStack Version:\nApache CloudStack 4.21.0.0\nDatabase:  10.6.22-MariaDB-0\nServer: ubuntu0.22.04.1-log Ubuntu 22.04\n\n### The steps to reproduce the bug\n\n1. Enable Usage Server in Apache CloudStack.\n2. Allow the Usage aggregation jobs to run normally for several days/weeks.\n3. Observe the usage_job table.\n4. Observe that successful jobs repeatedly start from the same historical date instead of advancing incrementally.\n5. Observe duplicate rows being inserted repeatedly into cloud_usage.\n6. Run:\nSELECT\n  vm_instance_id,\n  offering_id,\n  usage_type,\n  start_date,\n  end_date,\n  raw_usage,\n  COUNT(*) AS duplicates\nFROM cloud_usage\nWHERE usage_type = 13\nGROUP BY\n  vm_instance_id,\n  offering_id,\n  usage_type,\n  start_date,\n  end_date,\n  raw_usage\nHAVING COUNT(*) \u003e 1;\nObserve that duplicate counts increase daily.\n\n### What to do about it?\n\nPotential areas to investigate:\n\nUsage Server aggregation checkpoint handling\nUsage job state persistence\nReprocessing logic for historical aggregation windows\nNetwork Offering usage aggregation (usage_type=13)\nFailed usage_job recovery logic\nValidation for invalid aggregation windows where:\nstart_date \u003e end_date\nDeduplication protections before inserting into cloud_usage","author":{"url":"https://github.com/DaniloMurbach","@type":"Person","name":"DaniloMurbach"},"datePublished":"2026-05-06T23:20:04.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":4},"url":"https://github.com/13112/cloudstack/issues/13112"}