Repository navigation
url validator, exotic _wildcard_ issue #315
Description
Activity
$ python -c "from validators import __version__; print(__version__)" 0.22.0 $ python -c "from validators import url; print(url('hxxps://wildcard.jetsett.in/wwwmicrfofst/pohwernetsharaven8726skkksusukjs9916192629900177771017000172610720027111113'))" ValidationError(func=url, args={'value': 'hxxps://wildcard.jetsett.in/wwwmicrfofst/pohwernetsharaven8726skkksusukjs9916192629900177771017000172610720027111113'}) $ python -c "from validators import url; print(url('https://wildcard.jetsett.in/wwwmicrfofst/pohwernetsharaven8726skkksusukjs9916192629900177771017000172610720027111113'))" True
I do not understand, it works as expected.
hxxp/hxxpsis a convention, not a valid protocol.I get that hxxp/hxxps is just a convention. I only wanted to raise an issue about the http version.
The Markdown syntax in this issue removed the underscores from the url with
_wildcard_, sorry for that.import validators print(validators.url('https://_wildcard_.jetsett.in/wwwmicrfofst/pohwernetsharaven8726skkksusukjs9916192629900177771017000172610720027111113/'))When I run this I get this output:
ValidationError(func=url, args={'value': 'https://_wildcard_.jetsett.in/wwwmicrfofst/pohwernetsharaven8726skkksusukjs9916192629900177771017000172610720027111113/'})Hey @jeroengui the domain
_wildcard_.jetsett.inseems to be invalid.Because the following is valid according to
rfc_2782(TLDR: underscores are prepended not appended).$ python -c "from validators import url; print(url('https://_wildcard._jetsett.in/wwwmicrfofst/pohwernetsharaven8726skkksusukjs9916192629900177771017000172610720027111113', rfc_2782=True))" True
Are you sure it's not
_wildcard._jetsett.ininstead?I'm not sure about the standard and how more I read how more confused I get, but the thing is:
Most browsers are accepting urls with subdomains like
_wildcard_.jetsett.inand malicious actors are using them in phishing campaigns etc.At the moment it poses a real threat, because a lot of (antivirus) companies use validators for urls just like this one.
The urls never get in their blocklists, exposing their users to more risk because these malicious urls work fine in their browsers.I don't know if this is something that should be fixed in the browsers, in the standard or in the validators, and the issue is definitely bigger than just this implementation.
So to be clear:
We are figuring out if
http://_wildcard_.domain.comis a valid url.We agree that
http://_wildcard.jetsett.inandhttps://my_sarisari_store.typepad.com/are valid urls.So an underscore can be used in any place of the subdomain except the one right before the dot?
Stackoverflow has had some discussion on the topic also: https://stackoverflow.com/questions/2180465/can-domain-name-subdomains-have-an-underscore-in-it
EDIT: I added an example DNS record on my own domain that seems to work fine in most browsers : here (
https://_test_.jeroengui.be/)Some more information: It seems like SSL certificates cannot be generated for
_wildcard_.domain.combut a wildcard certificate like*.domain.comworks just fine for_wildcard_.domain.comhttps://cabforum.org/2018/11/12/ballot-sc12-sunset-of-underscores-in-dnsnames/
So appending underscore to sub/domain name is not encouraged, but it is technically valid.
This patch
- + r"(?:[a-zA-Z0-9-_]{0,61}[A-Za-z0-9])?\.)" + + rf"(?:[a-zA-Z0-9-_]{{0,61}}[A-Za-z0-9{'_'if rfc_2782 else ''}])?\.)"
in
validators/src/validators/domain.py
Line 45 in 854375f
will work.
You'll have to add
("example_.com", False, True)to pytest parameter list oftest_returns_true_on_valid_domain()intest_domain.py.PR is welcome.
I use the validators library in a URL scanning project. But this URL that work in the browser doesn't get recognized as a URL (replaced the tt by xx because it's a malicious sample):
hxxps://wildcard.jetsett.in/wwwmicrfofst/pohwernetsharaven8726skkksusukjs9916192629900177771017000172610720027111113/
I suppose the
_wildcard_part of the domain has something to do with it...