René's URL Explorer Experiment

Title: gh-91539: improve performance of get_proxies_environment by eendebakpt · Pull Request #91566 · python/cpython · GitHub

Open Graph Title: gh-91539: improve performance of get_proxies_environment by eendebakpt · Pull Request #91566 · python/cpython

X Title: gh-91539: improve performance of get_proxies_environment by eendebakpt · Pull Request #91566 · python/cpython

Description: Improve performance of get_proxies_environment when there are many environment variables. Improvements depend on the number of environment variables, but the method is several times faster in this PR. Fixes #91539 Performance test details Results: 0.1634683609008789 0.024187326431274414 with import os import time import urllib.request if 0: os.environ={ f'{ii}': 1 for ii in range(8000)} os.environ.update({ f'{ii}_proxy': 1 for ii in range(30)}) def getproxies_environment(): """Return a dictionary of scheme -> proxy server URL mappings. Scan the environment for variables named _proxy; this seems to be the standard convention. If you need a different way, you can pass a proxies dictionary to the [Fancy]URLopener constructor. """ # in order to prefer lowercase variables, process environment in # two passes: first matches any, second pass matches lowercase only # select only environment variables which end in (after making lowercase) _proxy candidate_names = [name for name in os.environ.keys() if len(name)>5 and name[-6]=='_'] # fast selection of candidates environment = [(name, os.environ[name], name.lower()) for name in candidate_names if name[-6:].lower()=='_proxy'] proxies = {} for name, value, name_lower in environment: if value and name_lower[-6:] == '_proxy': proxies[name_lower[:-6]] = value # CVE-2016-1000110 - If we are running as CGI script, forget HTTP_PROXY # (non-all-lowercase) as it may be set from the web server by a "Proxy:" # header from the client # If "proxy" is lowercase, it will still be used thanks to the next block if 'REQUEST_METHOD' in os.environ: proxies.pop('http', None) for name, value, name_lower in environment: if name[-6:] == '_proxy': if value: proxies[name_lower[:-6]] = value else: proxies.pop(name_lower[:-6], None) return proxies nn=400 t0=time.time() for ii in range(nn): urllib.request.getproxies_environment() dt=time.time()-t0 print(dt) t0=time.time() for ii in range(nn): getproxies_environment() dt=time.time()-t0 print(dt) Issue: gh-91539

Open Graph Description: Improve performance of get_proxies_environment when there are many environment variables. Improvements depend on the number of environment variables, but the method is several times faster in this ...

X Description: Improve performance of get_proxies_environment when there are many environment variables. Improvements depend on the number of environment variables, but the method is several times faster in this ...

Opengraph URL: https://github.com/python/cpython/pull/91566

X: @github

direct link

Domain: github.com

Links:

Viewport: width=device-width