Repository navigation
Respect system proxy exclusions. #1536
Description
Activity
- The windows registry
ProxyOverridefield.
A useful starting point is here...
proxyOverride = proxyOverride.split(';') # now check if we match one of the registry values. for test in proxyOverride: if test == '<local>': if '.' not in rawHost: return 1 test = test.replace(".", r"\.") # mask dots test = test.replace("*", r".*") # change glob sequence test = test.replace("?", r".") # change glob char
- The "exceptions" field returned by
from _scproxy import _get_proxy_settings()
On my system this returns...
>>> _get_proxy_settings() {'exclude_simple': False, 'exceptions': ('*.local', '169.254/16',)}
There's also an example documented in the
urllibsource code here...{ 'exclude_simple': bool, 'exceptions': ['foo.bar', '*.bar.com', '127.0.0.1', '10.1', '10.0/16']}So,
"10.0/16"and'169.254/16'here are not IPs, but IP ranges. Those are a bit awkward for us since we don't currently support subnet matching on transport mounts.- The windows registry
Thanks!
I think
httpxcan drop the support for system proxy settings, only uses environment settings:- It's hard to handle system proxy exclusions for
httpx's mounts system. - If you use the system proxy settings, you should check the system proxy-bypass settings, otherwise it's not correct.
Besides, for the field 'exceptions' ,
'10.1'='10.1/16'='10.1.1.1/16'.Ifhttpxneeds to mount system proxy-bypass settings on Windows and macosx, which can be unified into form like10.1.*, then callssocket.gethostbynameto check both hostname and ip for request url, or simply not callssocket.gethostbynamewhich means DNS lookups is not supported.- It's hard to handle system proxy exclusions for
Sticking to environment only settings would be one option, yes, though I'm not convinced that'd be the best from a user-experiance point of view.
Reacted by Asif Saif Uddin {"Auvi":"অভি"} and AccurioBeing able to pick up and use the system proxy settings — especially proxy auto config — would be a benefit in certain environments, even if it’s an option and not the default. I’m no longer in a situation like that, but Python PAC support on macOS could have made configuring some projects much easier. It might have been a reason to choose a library like HTTPX over another.
Reacted by Asif Saif Uddin {"Auvi":"অভি"}, aiudirog and RoDuthThis issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Still valid at the moment. Could do with a review and possibly extra docs.
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Upping the durations on you, @stalebot. Shoo.
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Nothing's changed in this topic i presume?
Lines 195 to 215 in db9072f
On Windows:
def getproxies(): return getproxies_environment() or getproxies_registry()
urllib.request.getproxies()reads proxies from environment variables first, then from Registry if environ variables are not set.getproxies_environment()can read proxy servers and exclusions from environment, butgetproxies_registry()can only read proxy servers from Registry.A possible solution for Windows is to rewrite
getproxies_registry(), read proxy servers and exclusions from Registry and return proxies in the form compatible with the return ofgetproxies_environment().This bites me right now. httpx picks up the proxy from
ProxyServerregistry key (for a local server which does not need it) and refuses to obeyProxyOverride. I am using a third party tool which internally uses httpx and I debugged deep into this library to figure this out.Reacted by vmahash-tataaigme too! It cost me so much time to find out why
httpx.getfailed whilerequests.getsucceed.This bites me right now. httpx picks up the proxy from
ProxyServerregistry key (for a local server which does not need it) and refuses to obeyProxyOverride. I am using a third party tool which internally uses httpx and I debugged deep into this library to figure this out.Finally, I solved it by manual declare the NO_PROXY environment:
export NO_PROXY="*.my-inner-domain.com;10.*;192.*;127.*"
We're currently leaning on
urllib.request.getproxies()to determine the system proxy setup, and setup which mounts should be a proxy transport and which should be a regular transport.However, we're not using
urllib.request.proxy_bypass(host).This all works as expected when environment settings are being used.
HTTP_PROXY,HTTPS_PROXY, andALL_PROXY. In that case we're readingNO_PROXY, and ensuring anything hostname patterns there are mounted as a regular transport...httpx/httpx/_utils.py
Lines 304 to 320 in 68cf1ff
However, in the case when none of those environment variables are set
getproxies()instead falls back to system proxy configuration. For windows this is registry based.ProxyEnableandProxyOverride. For Mac this is sysconf based.In those cases, we're correctly getting the configured proxies, but we aren't dealing with proxy exclusions.
We'd like to be able to setup these exclusions with our neat hostname pattern matched mounts system, which actually
means we can't just fallback to
urllib.request.proxy_bypass(host), because that needs to be called per-host.So, first steps...
ProxyOverridefield?from _scproxy import _get_proxy_settings()?