fix(bq_driver): Performance improvements SQLTables#1579
Merged
Conversation
shivamd-gpartner
force-pushed
the
SQLtables_optimization
branch
2 times, most recently
from
71d862c to
7742d6d
Compare
shivamd-gpartner
force-pushed
the
SQLtables_optimization
branch
from
7742d6d to
754544c
Compare
shivamd-gpartner
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
perf result after running on this pipeline
Now our driver is on par of the existing driver with SQLTables.
GetFilteredTables used to run a SELECT … FROM INFORMATION_SCHEMA.TABLES query job for every dataset. A BigQuery query job has multi-second fixed overhead regardless of result size. Switched it to the lightweight tables.list REST call + client-side name/type filtering — the same approach SQLColumns already used
When SQL_ATTR_METADATA_ID = TRUE, the catalog and schema arguments are exact identifiers, not patterns — yet the driver was calling projects.list (list all projects) and datasets.list (list all datasets) just to match one. That datasets.list call alone was 91 seconds on a busy project (throttled/retried). Now we use the names directly and skip both listings.