makedirs bug when multiprocessing

[simonm3]

I have multiple processes writing different files to the same folder. To ensure the folder hierarchy exists I call makedirs(recreate=True) before I create the files. Sometimes this fails saying folder already exists. This is because of race condition where another process created it after the check.

The pyfilesystem code in base/makedirs has thread lock which stops this happening in threads. It also has a try except to ignore the exception for the lowest subfolder. However this should be around the creation of parent folders too:

            dir_paths = tools.get_intermediate_dirs(self, path)
            for dir_path in dir_paths:
                self.makedir(dir_path, permissions=permissions)
            try:
                self.makedir(path)
            except errors.DirectoryExists:
                if not recreate:
                    raise

[simonm3]

Like this....also the code is missing a permissions parameter in the second call to makedir.

def makedirs(
    self,
    path,  # type: Text
    permissions=None,  # type: Optional[Permissions]
    recreate=False,  # type: bool
    ):
    self.check()
    with self._lock:
        dir_paths = tools.get_intermediate_dirs(self, path)
        for dir_path in dir_paths:
            try:
                self.makedir(dir_path, permissions=permissions)
            except errors.DirectoryExists:
                if not recreate:
                    raise
        try:
            self.makedir(path, permissions=permissions)
        except errors.DirectoryExists:
            if not recreate:
                raise
        return self.opendir(path)

[willmcgugan]

That looks sensible. Would you mind submitting a PR?

[willmcgugan]

Fixed in v2.4.9