urlunsplit for itms-services scheme returns invalid url

[AndyQ]

Relating to a Werkzueg issue (pallets/werkzeug#2691), when parsing an iOS App install url e.g.
itms-services:action=download-manifest&url=https://theacmeinc.com/abcdefeg, urlunpslit returns an invalid url.

e.g.

vals = urlparse( "itms-services://?action=download-manifest&url=https://theacmeinc.com/abcdefeg" )
print(vals)

newURL = urlunsplit((vals.scheme, vals.netloc, vals.path, vals.query, vals.params))
print( newURL )

prints:

ParseResult(scheme='itms-services', netloc='', path='', params='', query='action=download-manifest&url=https://theacmeinc.com/abcdefeg', fragment='')

itms-services:?action=download-manifest&url=https://theacmeinc.com/abcdefeg

Note the newURL is missing the // after the itms-services scheme.

This scheme is used to install ad-hoc and enterprise iOS apps.

Your environment

Tested on Apple M1 Max - 13.4 Beta (22F5049e)
Python: 3.10.10

For more details on the scheme here is a link to the Apple documentation (look for the "Use a website to distribute the app" section).
https://support.apple.com/en-gb/guide/deployment/depce7cefc4d/web

Linked PRs

• gh-104139: Add itms-services to uses_netloc urllib.parse. #104312

[davidism]

I don't think this scheme is standard. https://url.spec.whatwg.org/#url-serializing says "If url's host is non-null: Append "//" to output." This itms-services URL doesn't have a host, so it shouldn't require a //. I think there's some other special casing for schemes with the uses_netloc list, a similar "add //" could be added.

cpython/Lib/urllib/parse.py

Lines 44 to 74 in 9f9e001

	# A classification of schemes.
	# The empty string classifies URLs with no scheme specified,
	# being the default value returned by “urlsplit” and “urlparse”.
	uses_relative = ['', 'ftp', 'http', 'gopher', 'nntp', 'imap',
	'wais', 'file', 'https', 'shttp', 'mms',
	'prospero', 'rtsp', 'rtspu', 'sftp',
	'svn', 'svn+ssh', 'ws', 'wss']
	uses_netloc = ['', 'ftp', 'http', 'gopher', 'nntp', 'telnet',
	'imap', 'wais', 'file', 'mms', 'https', 'shttp',
	'snews', 'prospero', 'rtsp', 'rtspu', 'rsync',
	'svn', 'svn+ssh', 'sftp', 'nfs', 'git', 'git+ssh',
	'ws', 'wss']
	uses_params = ['', 'ftp', 'hdl', 'prospero', 'http', 'imap',
	'https', 'shttp', 'rtsp', 'rtspu', 'sip', 'sips',
	'mms', 'sftp', 'tel']
	# These are not actually used anymore, but should stay for backwards
	# compatibility. (They are undocumented, but have a public-looking name.)
	non_hierarchical = ['gopher', 'hdl', 'mailto', 'news',
	'telnet', 'wais', 'imap', 'snews', 'sip', 'sips']
	uses_query = ['', 'http', 'wais', 'imap', 'https', 'shttp', 'mms',
	'gopher', 'rtsp', 'rtspu', 'sip', 'sips']
	uses_fragment = ['', 'ftp', 'hdl', 'http', 'gopher', 'news',
	'nntp', 'wais', 'https', 'shttp', 'snews',
	'file', 'prospero']

[davidism]

@gpshead I know you were looking at some other urllib issues recently, could you comment on this?

[gpshead]

I agree with @davidism that the WhatWG URL spec does not require // when there is no pathname.

Regardless, behavior wise this seems to match our existing uses_netloc special case so we can just add it to that list in Lib/urllib/parse.py and add test coverage in Lib/test/test_urlparse.py. I made a PR.

Workaround for this to "work" on existing Pythons:

if "itms-services" not in urllib.parse.uses_netloc:
    urllib.parse.uses_netloc.append("itms-services")

I'm calling this a feature as code will have to deal with Python's that do not list it as such for a long time anyways via a code snippet like that, so backporting doesn't seem consistently helpful.

[gpshead]

fixed in 3.12.