René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"bogus usage jobs may be created when usage is behind in processing","articleBody":"### problem\n\nDuring trouble shooting missing data, when usage is behind it was found that a lot of usage_jobs are created that won’t have a period to process:\n\n```\n2026-01-13 01:00:38,566 DEBUG [cloud.usage.UsageManagerImpl] (Usage-HB-1:null) (logid:) it's been 86438558 ms since last usage job and 66261443 ms until next job, scheduling an immediate job to catch up (aggregation duration is 1440 minutes)\n2026-01-13 01:00:38,568 DEBUG [cloud.usage.UsageManagerImpl] (Usage-HB-1:null) (logid:) Scheduling Usage job...\n2026-01-13 01:00:38,570 INFO  [cloud.usage.UsageManagerImpl] (Usage-Job-1:null) (logid:) starting usage job...\n2026-01-13 01:00:38,606 INFO  [cloud.usage.UsageManagerImpl] (Usage-Job-1:null) (logid:) Parsing usage records between Mon Jan 12 01:00:00 CET 2026 and Tue Jan 13 00:59:59 CET 2026\n2026-01-13 01:01:38,581 DEBUG [cloud.usage.UsageManagerImpl] (Usage-HB-1:null) (logid:) it's been 86498581 ms since last usage job and 66201420 ms until next job, scheduling an immediate job to catch up (aggregation duration is 1440 minutes)\n2026-01-13 01:01:38,581 DEBUG [cloud.usage.UsageManagerImpl] (Usage-HB-1:null) (logid:) Scheduling Usage job...\n2026-01-13 01:02:38,559 DEBUG [cloud.usage.UsageManagerImpl] (Usage-HB-1:null) (logid:) it's been 86558559 ms since last usage job and 66141442 ms until next job, scheduling an immediate job to catch up (aggregation duration is 1440 minutes)\n2026-01-13 01:02:38,559 DEBUG [cloud.usage.UsageManagerImpl] (Usage-HB-1:null) (logid:) Scheduling Usage job...\n2026-01-13 01:03:38,558 DEBUG [cloud.usage.UsageManagerImpl] (Usage-HB-1:null) (logid:) it's been 86618558 ms since last usage job and 66081443 ms until next job, scheduling an immediate job to catch up (aggregation duration is 1440 minutes)\n2026-01-13 01:03:38,558 DEBUG [cloud.usage.UsageManagerImpl] (Usage-HB-1:null) (logid:) Scheduling Usage job...\n2026-01-13 01:04:38,559 DEBUG [cloud.usage.UsageManagerImpl] (Usage-HB-1:null) (logid:) it's been 86678559 ms since last usage job and 66021442 ms until next job, scheduling an immediate job to catch up (aggregation duration is 1440 minutes)\n2026-01-13 01:04:38,559 DEBUG [cloud.usage.UsageManagerImpl] (Usage-HB-1:null) (logid:) Scheduling Usage job...\n2026-01-13 01:05:38,558 DEBUG [cloud.usage.UsageManagerImpl] (Usage-HB-1:null) (logid:) it's been 86738559 ms since last usage job and 65961442 ms until next job, scheduling an immediate job to catch up (aggregation duration is 1440 minutes)\n2026-01-13 01:05:38,558 DEBUG [cloud.usage.UsageManagerImpl] (Usage-HB-1:null) (logid:) Scheduling Usage job...\n2026-01-13 01:06:02,952 DEBUG [cloud.usage.UsageManagerImpl] (Usage-Job-1:null) (logid:) Creating networking offering: 18 for Vm: 75432 for account: 239496\n2026-01-13 01:06:03,028 DEBUG [cloud.usage.UsageManagerImpl] (Usage-Job-1:null) (logid:) create volume with id : 163459 for account: 239496\n2026-01-13 01:06:03,497 DEBUG [cloud.usage.UsageManagerImpl] (Usage-Job-1:null) (logid:) deleting network offering: 18 from Vm: 96981\n```\nthe HeartBeat thread should be clever about existing jobs and not create the extra jobs.\n\n\n### versions\n\nACS 4.19\n\n### The steps to reproduce the bug\n\n1. have an environment running for a while\n2. prune intermediate and usage tables a couple of months\n3. set the events of those months to unprocessed\n4. start usage\n5. observe the a job starts and multiple other jobs are created.\n6. the extra jobs seem to have an interval of -1 second and fail (of course)\n\n\n### What to do about it?\n\nmake sure the heartbeat job does not create jobs while jobs are being processed/scheduled already.","author":{"url":"https://github.com/DaanHoogland","@type":"Person","name":"DaanHoogland"},"datePublished":"2026-01-14T12:40:02.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":3},"url":"https://github.com/12424/cloudstack/issues/12424"}