René's URL Explorer Experiment

Title: gh-101178: refactor base64.b85encode to be memory friendly by romuald · Pull Request #112248 · python/cpython · GitHub

Open Graph Title: gh-101178: refactor base64.b85encode to be memory friendly by romuald · Pull Request #112248 · python/cpython

X Title: gh-101178: refactor base64.b85encode to be memory friendly by romuald · Pull Request #112248 · python/cpython

Description: Current description Rewrote the base64._85encode method logic in C, by plugging in the binascii module (already taking care of the bae64 methods) By using C and a single buffer, the memory use is reduced to a minimum, addressing the initial issue. It also greatly improves performance as a bonus: main SMALL (11 bytes): 1575427 iterations (1.27 µs per call, 115.41 ns per byte) MEDIUM (200 bytes): 204909 iterations (9.76 µs per call, 48.80 ns per byte) BIG (5000 bytes): 8623 iterations (231.94 µs per call, 46.39 ns per byte) VERYBIG (500000 bytes): 81 iterations (24.69 ms per call, 49.38 ns per byte) branch SMALL (11 bytes): 11230718 iterations (178.08 ns per call, 16.19 ns per byte) MEDIUM (200 bytes): 6004721 iterations (333.07 ns per call, 1.67 ns per byte) BIG (5000 bytes): 458005 iterations (4.37 µs per call, 873.35 ps per byte) VERYBIG (500000 bytes): 4772 iterations (419.11 µs per call, 838.22 ps per byte) Script used to test: https://gist.github.com/romuald/7aeba5f40693bb351da4abe62ad7321d Previous description (python refactor) not up to date with current PR Refactor code to make use of generators instead of allocating 2 potentially huge lists for large datasets Memory gain only measured using macOS and a 5Mb input. Using main: Before encoding Physical footprint: 16.3M Physical footprint (peak): 21.3M After encoding Physical footprint: 45.0M Physical footprint (peak): 244.1M With refactor: Before encoding Physical footprint: 14.6M Physical footprint (peak): 19.6M After encoding Physical footprint: 28.5M Physical footprint (peak): 34.4M The execution time is more than doubled, which may not be acceptable. However the memory used is reduced by more than 90% edit: changed the algorithm to be more efficient, the performance decrease now seems to be negligible I also have no idea how (and if) I should test this Here is the script I've used to measure the execution time, the memdebug can probably be adapted to read /proc/{pid} on Linux edit: updated to work on Linux too import os import sys import random import hashlib import platform import subprocess from time import time from base64 import b85encode def memdebug(): if platform.system == "Darwin": if not os.environ.get("MallocStackLogging"): return res = subprocess.check_output(["malloc_history", str(os.getpid()), "-highWaterMark", "-allBySize"]) for line in res.splitlines(): if line.startswith(b"Physical"): print(line.decode()) elif platform.system() == "Linux": with open(f"/proc/{os.getpid()}/status") as reader: for line in reader: if line.startswith("VmPeak:"): print(line, end="") def main(): # use a stable input rnd = random.Random() rnd.seed(42) data = rnd.randbytes(5_000_000) memdebug() start = time() import pdb try: res = b85encode(data) except Exception: # pdb.post_mortem() raise end = time() memdebug() print("Data length:", len(data)) print("Output length:", len(res)) print(f"Decode time: {end-start:.3f}s") h = hashlib.md5(res).hexdigest() print("Hashed result", h) assert h == "ad97e45ba085865e70f7aa05c9a31388" if __name__ == '__main__': main() Issue: gh-101178

Open Graph Description: Current description Rewrote the base64._85encode method logic in C, by plugging in the binascii module (already taking care of the bae64 methods) By using C and a single buffer, the memory use is r...

X Description: Current description Rewrote the base64._85encode method logic in C, by plugging in the binascii module (already taking care of the bae64 methods) By using C and a single buffer, the memory use is r...

Opengraph URL: https://github.com/python/cpython/pull/112248

X: @github

direct link

Domain: github.com

Links:

Viewport: width=device-width