René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Aggregate expired pid db files, control the number of files and improve scrape effectiveness","articleBody":"history related issues: https://github.com/prometheus/client_python/pull/441 \r\n https://github.com/prometheus/client_python/pull/430\r\n\r\nCan we aggregate all the db files from a period of time ago and non-current pid into a total db file, to control the number of pid files ？\r\n\r\nI have realized this idea with golang，here are some details：\r\n\r\n**Project deploy info：  \r\ngunicorn django  \r\n128 workers  \r\ngunicorn max_requests：10000（create a new pid file almost every minute）**\r\n\r\n\r\n1. I can't solve the problem that the pid file has been growing, and it can reach 6,000 in four days；\r\n2. Try to delete the expired pid regularly in the code, but it will cause the figure to drop with grafana；\r\n3. The time to request metric is getting longer as the program runs.\r\n\r\n\r\nImprove scrape efficiency:\r\nI used golang to rewrite the logic of python aggregate metrics（generate metric still using python）. After rewriting, each scrape time is less than 1 second. \r\n\r\nSolve the growing pid files:\r\nAggregate all the db files from a period of time ago and non-current pid into a total db file. Then delete these files. When calculating metric, history total db + curent pid = current pid db. (i do it every hour)\r\n\r\nNow, num of pid files is \u003c200 in my project. if we can do change this, it would be a big strengthen. Just like prometheus will also aggregate historical data","author":{"url":"https://github.com/cai-personal","@type":"Person","name":"cai-personal"},"datePublished":"2019-07-26T05:23:27.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":1},"url":"https://github.com/443/client_python/issues/443"}