René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"beaver dramatically falls behind watched logfile ","articleBody":"I have a log file that is updated about 60 times per second with big lines, about 160 KB per line. I increased read chunk size from 4096 to 40960 (10 times, file tail.py, method _run_pass), beaver read new lines faster, but not enough. The lag between last log entry and line processing by beaver increases -  10 seconds delay per minute of working. \r\nI've tested with tranport type = stdout, format: raw\r\n\r\nIncreasing  number_of_consumer_processes to 2 or 4 doesn't help.\r\nIncreasing reading chunk size to 500 KB doesn't help.\r\n\r\nIt is not a disk issue (SSDin this server and some stat about IO operations in Grafana looks good).\r\n\r\nWhy beaver reading lines from log can be so slow?\r\nAny hints how speed it up?\r\nCan increasing discover_interval help?","author":{"url":"https://github.com/nburunova","@type":"Person","name":"nburunova"},"datePublished":"2018-11-09T10:34:27.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":0},"url":"https://github.com/432/python-beaver/issues/432"}