René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"performance: Update io.DEFAULT_BUFFER_SIZE to make python IO faster?","articleBody":"# Bug report\n\n### Bug description:\n\nHello, \n\nI was doing some benchmarking of python and package installation.\nThat got me down a rabbit hole of buffering optimizations between between pip, requests, urllib and the cpython interpreter.\n\nTL;DR I would like to discuss updating the value of io.DEFAULT_BUFFER_SIZE. It was set to 8192 since 16 years ago.\noriginal commit: https://github.com/python/cpython/blame/main/Lib/_pyio.py#L27\n\nIt was a reasonable size given hardware and OS at the time. It's far from optimal today.\nRemember, in 2008 you'd run a 32 bits operating system with less than 2 GB memory available and to share between all running applications. \nBuffers had to be small, few kB, it wasn't conceivable to have buffer measured in entire MB.\n\nI will attach benchmarks in the next messages showing 3 to 5 times write performance improvement when adjusting the buffer size.\n\nI think the python interpreter can adopt a buffer size somewhere between 64k to 256k by default.\nI think 64k is the minimum for python and it should be safe to adjust to.\nHigher is better for performance in most cases, though there may be some cases where it's unwanted \n(seek and small read/writes, unwanted trigger of write ahead, slow devices with throughput in measured in kB/s where you don't want to block for long)\n\nIn addition, I think there is a bug in open() on Linux.\nopen() sets the buffer size to the device block size on Linux when available (st_blksize, 4k on most disks), instead of io.DEFAULT_BUFFER_SIZE=8k.\nI believe this is unwanted behavior, the block size is the minimal size for IO operations on the IO device, it's not the optimal size and it should not be preferred.\nI think open() on Linux should be corrected to use a default buffer size of `max(st_blksize, io.DEFAULT_BUFFER_SIZE)` instead of `st_blksize`?\n\nRelated, the doc might be misleading for saying st_blksize is the preferred size for efficient I/O. https://github.com/python/cpython/blob/main/Doc/library/os.rst#L3181\nThe GNU doc was updated to clarify: \"This is not guaranteed to give optimum performance\" https://www.gnu.org/software/gnulib/manual/html_node/stat_002dsize.html\n\nThoughts?\n\n\n\nAnnex: some historical context and technical considerations around buffering.\n\nOn the hardware side:\n* HDD had 512 bytes blocks historically, then HDD moved to 4096 bytes blocks in the 2010s.\n* SSD have 4096 bytes blocks as far as I know.\n\nOn filesystems:\n* buffer size should never be smaller than device and filesystem blocksize\n* I think ext3, ext4, xfs, ntfs, etc... follow the device block size of 4k as default, though they can be configured for any block size.\n* NTFS is capped to 16TB maximum disk size with 4k blocks.\n* microsoft recommends 64k block size for windows server 2019+ and larger disks https://learn.microsoft.com/en-us/windows-server/storage/file-server/ntfs-overview\n* RAID setups and assimilated with zfs/btrfs/xfs can have custom block size, I think anywhere 4kB-1MB. I don't know if there is any consensus, I think anything 16k-32k-64k-128k can be seen in the wild.\n\nOn network filesystems:\n* shared network home directories are common on linux (NFS share) and windows (SMB share).\n* entreprise storage vendors like Pure/Vast/NetApp recommend 524488 or 1048576 bytes for IO. \n* see rsize wsize in mount settings: \n* `host:path on path type nfs (rw,relatime,vers=3,rsize=1048576,wsize=1048576,acregmin=60,acdirmin=60,hard,proto=tcp,nconnect=8,mountproto=tcp, ...)`\n* for windows I cannot find documentation for network clients, though the windows server should have the NTFS filesystem with at least 64k block size as per microsoft recommendation above.\n\nOn pipes:\n* buffering is used by pipes and for interprocess communications. see subprocess.py\n* posix guarantees that writes to pipes are atomic up to PIPE_BUF, 4096 bytes on Linux kernel, guaranteed to be at least 512 bytes by posix.\n* Python had a default of io.DEFAULT_BUFFER_SIZE=8192 so it never benefitted from that atomic property :D\n\non compression code, they probably all need to be adjusted:\n* the buffer size is used by compression code in cpython: gzip.py lzma.py bz2.py\n* I think lzma and bz2 are using the default size.\n* gzip is using a 128kb read buffer, somebody realized it was very slow 2 years ago and rewrote the buffering to 128k.\n* then somebody else realized last year it was still very slow to write and added an arbitrary write buffer 4*io.DEFAULT_BUFFR_SIZE.\n* https://github.com/python/cpython/commit/eae7dad40255bad42e4abce53ff8143dcbc66af5\n* https://github.com/python/cpython/issues/89550\n* base64 is reading in chunks of 76 characters???\n* https://github.com/python/cpython/blob/main/Lib/base64.py#L532\n\nOn network IO:\n* On Linux, TCP read and write buffers were a minimum of 16k historically. The read buffer was increased to 64k in kernel v4.20, year 2018\n* the buffer is resized dynamically with the TCP window upto 4MB write 6M read, let's not get into TCP.  see sysctl_tcp_rmem sysctl_tcp_wmem\n* linux code: https://github.com/torvalds/linux/blame/master/net/ipv4/tcp.c#L4775\n* commit Sep 2018: https://github.com/torvalds/linux/commit/a337531b942bd8a03e7052444d7e36972aac2d92\n* I think socket buffers are managed separately by the kernel, the io.DEFAULT_BUFFER_SIZE matters when you read a file and write to network, or read from network and write to file.\n\n\non HTTP, a large subset of networking:\n* HTTP is large file transfer and would benefit from a much larger buffer, but that's probably more of a concern for urllib/requests. \n* requests.content is 10k chunk by default.\n* requests iter_lines(chunk_size=512, decode_unicode=False, delimiter=None) is 512 chunk by default.\n* requests iter_content(chunk_size=1, decode_unicode=False) is 1 byte by default\n* source: set in 2012 https://github.com/psf/requests/blame/8dd3b26bf59808de24fd654699f592abf6de581e/src/requests/models.py#L80\n\nnote to self: remember to publish code and result in next message\n\n\n### CPython versions tested on:\n\n3.11\n\n### Operating systems tested on:\n\nOther\n\n\u003c!-- gh-linked-prs --\u003e\n### Linked PRs\n* gh-118037\n* gh-118144\n* gh-119783\n* gh-131052\n\u003c!-- /gh-linked-prs --\u003e\n","author":{"url":"https://github.com/morotti","@type":"Person","name":"morotti"},"datePublished":"2024-03-22T11:41:23.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":11},"url":"https://github.com/117151/cpython/issues/117151"}