René's URL Explorer Experiment

Title: bpo-29870: ssl socket leak by misg · Pull Request #981 · python/cpython · GitHub

Open Graph Title: bpo-29870: ssl socket leak by misg · Pull Request #981 · python/cpython

X Title: bpo-29870: ssl socket leak by misg · Pull Request #981 · python/cpython

Description: I'm using Python 3.6 and aiohttp 2.0.5. Given this server: import http.server import ssl addr = ('localhost', 8081) httpd = http.server.HTTPServer(addr, http.server.SimpleHTTPRequestHandler) httpd.socket = ssl.wrap_socket(httpd.socket, keyfile='server.key', certfile='server.cert', server_side=True) httpd.serve_forever() and this client (that basically sends get requests to the above server): import asyncio import aiohttp import resource last = 0 def handler(*_): global last mem = resource.getrusage(resource.RUSAGE_SELF).ru_maxrss / 1024 print('Variation:', mem - last) last = mem async def get(session): response = await session.request('get', 'https://0.0.0.0:8081') # https (ssl) print(response.status) response.close() async def main(loop): conn = aiohttp.TCPConnector(verify_ssl=False, enable_cleanup_closed=True) session = aiohttp.ClientSession(loop=loop, connector=conn) for x in range(500): await get(session) await asyncio.sleep(1, loop=loop) handler() session.close() if __name__ == '__main__': loop = asyncio.get_event_loop() loop.run_until_complete(main(loop)) I have been able to reproduce the issue. In fact, every 6-7 get requests, I got a Variation: 0.25 printed on my screen that indicates my client uses 0.25 more MB of memory (it just grows indefinitely, I tested it with 10000 requests). Using ipdb and objgraph led me to this graph where you can clearly see that there are some circular references between SSLProtocol and _SSLPipe objects: I know that since PEP 442 it shouldn't be a problem anymore, the python garbage collector doesn't care about the cycle and collect them anyway. But the interesting fact is that breaking this cycle (see my commit) actually fixes the memory leak in my client (no more Variation: 0.25, just always Variation: 0.0). My hypothesis is that circular references between SSLProtocol and _SSLPipe actually hide some circular references between some C objects (that define some tp_dealloc slots), that lead to these memory leaks.

Open Graph Description: I'm using Python 3.6 and aiohttp 2.0.5. Given this server: import http.server import ssl addr = ('localhost', 8081) httpd = http.server.HTTPServer(addr, http.server.SimpleHTTPRequestHa...

X Description: I'm using Python 3.6 and aiohttp 2.0.5. Given this server: import http.server import ssl addr = ('localhost', 8081) httpd = http.server.HTTPServer(addr, http.server.SimpleH...

Opengraph URL: https://github.com/python/cpython/pull/981

X: @github

direct link

Domain: github.com

Links:

Viewport: width=device-width