Fix s3 sync filters with absolute path patterns

[Om-singhaI]

Issue #, if available: #8932

Description of changes:

Root cause

Filter.__init__ in awscli/customizations/s3/filters.py rebases every --exclude and --include pattern onto the source root and the destination root with os.path.join. For an absolute pattern, os.path.join returns the pattern unchanged and silently drops the root, so the destination copy of the pattern stays a local path that can never match a destination path of the form bucket/key. sync builds two create_filter instances in subcommands.py and the second one filters the destination listing, which is where the dropped root bites. @mapk-amazon's first comment pointed at filters.py; this join is the exact mechanism.

Reproduction

The filters do look "consistent between using absolute path and relative path" on the source side, and the ListObjectsV2 response is correct; the divergence is only in the destination copy of the patterns. It is visible at the filter level with no credentials:

from awscli.customizations.s3.filegenerator import FileStat
from awscli.customizations.s3.filters import create_filter

def stat(path, src_type):
    return FileStat(src=path, dest='', compare_key='', size=10,
                    last_update=0, src_type=src_type,
                    dest_type='s3' if src_type == 'local' else 'local',
                    operation_name='')

# aws s3 sync /tmp/src s3://bucket/ --exclude "*" --include "/tmp/src/photos/*"
f = create_filter({'filters': [['--exclude', '*'],
                               ['--include', '/tmp/src/photos/*']],
                   'dir_op': True, 'src': '/tmp/src', 'dest': 's3://bucket/'})
print('dst_patterns:', f.dst_patterns)
print('local listing:',
      [x.src for x in f.call([stat('/tmp/src/photos/cat.jpg', 'local')])])
print('remote listing:',
      [x.src for x in f.call([stat('bucket/photos/cat.jpg', 's3')])])

# aws s3 sync /tmp/src s3://bucket/ --delete --exclude "/tmp/src/keep/*"
f2 = create_filter({'filters': [['--exclude', '/tmp/src/keep/*']],
                    'dir_op': True, 'src': '/tmp/src', 'dest': 's3://bucket/'})
print('protected object in remote listing:',
      [x.src for x in f2.call([stat('bucket/keep/important.txt', 's3')])])

On develop:

dst_patterns: [('exclude', 'bucket/*'), ('include', '/tmp/src/photos/*')]
local listing: ['/tmp/src/photos/cat.jpg']
remote listing: []
protected object in remote listing: ['bucket/keep/important.txt']

The relative exclude rebased to bucket/* and removed every remote object; the absolute include could not rebase, so nothing was put back. The Comparator then sees an empty remote listing and uploads every included file on every run, so the sync never converges. The last line is the --delete case: the object the absolute --exclude was written to protect stays in the destination listing and the mirror deletes it.

With this change, an absolute pattern under a local root is reduced to its relative form before rebasing, so it applies to both sides exactly as the relative spelling does:

dst_patterns: [('exclude', 'bucket/*'), ('include', 'bucket/photos/*')]
local listing: ['/tmp/src/photos/cat.jpg']
remote listing: ['bucket/photos/cat.jpg']
protected object in remote listing: []

Changes

• awscli/customizations/s3/filters.py: strip the local root prefix from an absolute pattern before rebasing. The prefix check is textual so wildcard characters stay as typed, and compares with os.path.normcase to handle drive letter casing on Windows. Absolute patterns under neither root are left untouched.
• tests/unit/customizations/s3/test_filters.py: six new cases covering both sides of an absolute include, exclude protection under --delete, downloads with a pattern under the destination root, a pattern outside any root, a root with a trailing separator, and Windows drive letter casing exercised through ntpath on every platform. The existing cases only pin relative patterns.
• tests/functional/s3/test_sync_command.py: test_sync_with_absolute_include_converges models the second run of the reporter's command and asserts the only request is ListObjectsV2, with zero PutObject calls.
• .changes/next-release/bugfix-s3sync-8932.json: changelog entry.

Testing

$ python -m pytest tests/unit/customizations/s3/
525 passed
$ python -m pytest tests/functional/s3/
245 passed, 14 skipped

With only filters.py reverted, six of the seven new tests fail; the seventh pins the untouched handling of patterns outside any root.

Relative patterns take the same code path as before, and s3 roots are never absolute paths, so cp, mv, rm and s3 to s3 transfers behave exactly as before unless a pattern is an absolute path under a local root.

Fixes #8932

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.