BigQuery properties issues in query.py

(1) When I run a long, non-cached, sync query --

query_job = bq_client.run_sync_query(query_string)
query_job.run()

... it often returns without being complete. I call query_job.fetch_data(page_token=None) to try and get the result. If the query finished by that time -- all is well, however if doesn't, the server response doesn't include the job's name (jobReference.jobId); since the state is being completely erased/rewritten, following calls to fetch_data fail with ValueError("Query not yet executed: call 'run()'").

(2) The run method checks if self._job is not None: to prevent a double invocation. It should however run that check with self.job instead (no underscore), as self._job is only set if you previously used self.job.

(3) It would be 'nicer' if the total_rows property return an integer (or None) rather than a string.

@amirshavit

@amirshavit Thanks for the report! I've set the back-end label because the API docs for jobs/getQueryResults say that the resource returned from that API must include the jobReference.jobId subkey.

I've split out your second and third points into separate issues (~~#3002~~, ~~#3003~~, ~~#3004~~).

I will reroute this to the BigQuery folks.

@tswast

/cc @tswast

The OP is correct that these are issues in the design of the current library API. We're working on a revised version that will make the long-running query flow more convenient to execute.

What's the status of this? Does 0.26.x fix the problem?

Seems like it's not fixed; almost 6 months 😞

The redesign will not have a separate "sync" query versus "async" query. To get the behavior of what the query method will be, use the async query method. With the 0.26.0 release the QueryJob class follows the PEP 3148 Futures API.

    import uuid
    from google.cloud import bigquery

    client = bigquery.Client()
    query_job = client.run_async_query(str(uuid.uuid4()), query)
    query_job.begin()

    query_job.result()  # Wait for job to complete
    destination_table = query_job.destination
    destination_table.reload()
    for row in destination_table.fetch_data():
        print(row)

@tswast

@tswast Closing this as your redesign supercedes it. Reopen if you disagree.

chore: move samples from python-docs-sample (#66)

Navigation Menu

Repository navigation

BigQuery properties issues in query.py #3001

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

BigQuery properties issues in query.py #3001

Description

Activity

tseaver commented on Feb 13, 2017

lukesneeringer commented on Feb 28, 2017

tseaver commented on May 16, 2017

jcondit commented on Jun 5, 2017

aschmolck commented on Jul 14, 2017

amirshavit commented on Aug 8, 2017

tswast commented on Aug 8, 2017

lukesneeringer commented on Aug 11, 2017

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions