chore: add test sharding to unit tests

[daniel-sanche]

Added test sharding for unit and mypy tests:

• added initialize job to unit test CI. It analyzes the modified packages, and spawns up to MAX_SHARDS jobs for each supported Python runtime
• long-running packages can have a multiplier added in PACKAGE_WEIGHTS. This will cause the system to reserve more time for long-running tests, to better distribute the workload across shards
• shards are added according to the formula `shards_used = min(1, max(MAX_SHARDS, total_package_weight / 10))
• adds an end unit test complete step, which is only green if all shards pass. This can be our new required check for unit tests
• if any single unit test shard fails, the remaining end early to provide quick feedback
• preview packages will also be selected when modified, and put in the same shard as their parent package (for easier discovery)

Before merge:

• we need to update the required checks. We can do this after approval, but before merging

[gemini-code-assist]

Code Review

This pull request introduces parallel execution for system tests and implements a sharding mechanism for CI jobs, including a new Python script to group packages and updates to the test runner script. Feedback on these changes focuses on improving reliability and safety: first, by using an EXIT trap in .kokoro/system.sh to guarantee cleanup of isolated gcloud configuration directories in case of test failures; and second, by avoiding global toggles of set -e in ci/run_conditional_tests.sh and instead capturing test exit codes using the || operator.

[parthea]

Added suggestions to satisfy findings in https://github.com/googleapis/google-cloud-python/actions/runs/30566874706/job/90954041456 from PR https://github.com/googleapis/google-cloud-python/pulls