René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"High cpu when using Gunicorn with multiprocess","articleBody":"I run flask app on Gunicorn 20.0.4 and Kubernetes. My python is v3.7.5. I have a problem with increasing CPU over a time.\r\n![image](https://user-images.githubusercontent.com/29543308/88379047-7e903f00-cda2-11ea-8fbf-0e5b22b082a8.png)\r\nAfter investigating stack traces using py-spy I noticed that the issue is caused by prometheus metrics lib.\r\nIt doesn't clean up metric files from old workers. Therefore over a time merge process take more and more CPU. \r\n\r\nAfter deleting following files the CPU usages dropped significantly:\r\n```\r\nll -lh\r\ntotal 7.0M\r\ndrwxr-xr-x 2 root root 4.0K Jul 24 06:13 ./\r\ndrwxrwxrwt 1 root root 4.0K Jul 19 14:14 ../\r\n-rw-r--r-- 1 root root 1.0M Jul 24 08:14 counter_106.db\r\n-rw-r--r-- 1 root root 1.0M Jul 23 18:41 counter_112.db\r\n-rw-r--r-- 1 root root 1.0M Jul 24 04:07 counter_118.db\r\n-rw-r--r-- 1 root root 1.0M Jul 24 04:54 counter_136.db\r\n-rw-r--r-- 1 root root 1.0M Jul 24 08:40 counter_142.db\r\n-rw-r--r-- 1 root root 1.0M Jul 20 16:44 counter_16.db\r\n-rw-r--r-- 1 root root 1.0M Jul 20 11:24 counter_17.db\r\n-rw-r--r-- 1 root root 1.0M Jul 21 01:40 counter_18.db\r\n-rw-r--r-- 1 root root 1.0M Jul 21 20:14 counter_40.db\r\n-rw-r--r-- 1 root root 1.0M Jul 21 17:17 counter_52.db\r\n-rw-r--r-- 1 root root 1.0M Jul 21 21:29 counter_58.db\r\n-rw-r--r-- 1 root root 1.0M Jul 23 07:19 counter_70.db\r\n-rw-r--r-- 1 root root 1.0M Jul 22 19:49 counter_82.db\r\n-rw-r--r-- 1 root root 1.0M Jul 22 18:59 counter_88.db\r\n-rw-r--r-- 1 root root 1.0M Jul 24 08:43 histogram_106.db\r\n-rw-r--r-- 1 root root 1.0M Jul 24 04:15 histogram_112.db\r\n-rw-r--r-- 1 root root 1.0M Jul 24 05:02 histogram_118.db\r\n-rw-r--r-- 1 root root 1.0M Jul 24 08:43 histogram_136.db\r\n-rw-r--r-- 1 root root 1.0M Jul 24 08:43 histogram_142.db\r\n-rw-r--r-- 1 root root 1.0M Jul 20 16:46 histogram_16.db\r\n-rw-r--r-- 1 root root 1.0M Jul 20 11:45 histogram_17.db\r\n-rw-r--r-- 1 root root 1.0M Jul 21 01:51 histogram_18.db\r\n-rw-r--r-- 1 root root 1.0M Jul 21 22:41 histogram_40.db\r\n-rw-r--r-- 1 root root 1.0M Jul 21 17:45 histogram_52.db\r\n-rw-r--r-- 1 root root 1.0M Jul 22 01:44 histogram_58.db\r\n-rw-r--r-- 1 root root 1.0M Jul 23 07:37 histogram_70.db\r\n-rw-r--r-- 1 root root 1.0M Jul 23 01:01 histogram_82.db\r\n-rw-r--r-- 1 root root 1.0M Jul 22 23:40 histogram_88.db\r\n```\r\n![image](https://user-images.githubusercontent.com/29543308/88379256-e21a6c80-cda2-11ea-8836-b27f3281c5de.png)\r\n\r\nThe issue is related to https://github.com/prometheus/client_python/issues/275\r\n\r\nCan we avoid that somehow without periodically restating the k8s pod? Maybe multiprocess should use PID + UUID generated on worker rather than just PID for file names? So master could remove/merge files from dead workes?\r\n\r\n","author":{"url":"https://github.com/onzo-mateuszzakarczemny","@type":"Person","name":"onzo-mateuszzakarczemny"},"datePublished":"2020-07-24T09:46:21.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":14},"url":"https://github.com/568/client_python/issues/568"}