GH-41960: [Python] Expose new S3 option check_directory_existence_before_creation

[HaochengLIU]

Rationale for this change

Expose new S3 option check_directory_existence_before_creation from GH-41493

What changes are included in this PR?

Expose new S3 option check_directory_existence_before_creation from GH-41493

Are these changes tested?

yes

Are there any user-facing changes?

Yes. Python function documentation is updated.

• GitHub Issue: [Python] Expose new S3 option check_directory_existence_before_creation #41960

[github-actions]

⚠️ GitHub issue #41960 has been automatically assigned in GitHub to PR creator.

[HaochengLIU]

@pitrou Here is the Python side exposure PR
I have very limited knowledge about CPython as I normally use Pybind11, I just follow the existing code for allow_bucket_deletion
Please review when you have cycle 😃
ty

[jorisvandenbossche]

The code looks good! Just some comments on the docstring

[jorisvandenbossche]

Suggested change

	check_directory_existence_before_creation: boolean, default false
	check_directory_existence_before_creation : boolean, default false

[jorisvandenbossche]

It's rather unclear what "pessimistic" means in this context (but I see this term is used on the C++ side as well). Also, we shouldn't mention "CreateDir" as that is a C++ function the python user is not familiar with. We can use a generic term, something like "when creating a directory"

[jorisvandenbossche]

Suggested change

	if false, then CreateDir will try to create the directory without checking its
	By default (False), when creating a directory, it will try to create the directory without checking its

I would maybe also include the "It's an optimization to try directory creation and catch the error, rather than issue two dependent I/O calls." as is explained in C++

[HaochengLIU]

@jorisvandenbossche thanks for the suggestions, please review again

[github-actions]

⚠️ GitHub issue #41960 has been automatically assigned in GitHub to PR creator.

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 6 benchmarking runs that have been run so far on merge-commit b51e997.

There were 8 benchmark results indicating a performance regression:

• Commit Run on ursa-i9-9960x at 2024-06-11 02:36:31Z
  • file-read (R) with compression=lz4, dataset=nyctaxi_2010-01, file_type=feather, language=R, output_type=dataframe
  • file-read (R) with compression=uncompressed, dataset=nyctaxi_2010-01, file_type=feather, language=R, output_type=dataframe
• and 6 more (see the report linked below)

The full Conbench report has more details. It also includes information about 7 possible false positives for unstable benchmarks that are known to sometimes produce them.