BigQuery: list_jobs() iterator encounters error when calling classmethod from_api_repr()

[sean185]

Disclaimer: I am new to opening issues so please guide me along on what I can do to improve, thanks.

1. Ubuntu 14.04

2. Python 2.7.6

3. google-cloud-python version

google-api-python-client==1.5.5
google-auth==0.4.0
google-auth-httplib2==0.0.2
google-cloud-bigquery==0.22.0
google-cloud-core==0.22.0
googleapis-common-protos==1.5.0

4. Stacktrace if available

Traceback (most recent call last):
  File "pybq_test.py", line 103, in <module>
    main()
  File "pybq_test.py", line 69, in main
    for i,job in enumerate(jobs):
  File "/home/dev-sean/BQTesting/local/lib/python2.7/site-packages/google/cloud/iterator.py", line 211, in _items_iter
    for item in page:
  File "/home/dev-sean/BQTesting/local/lib/python2.7/site-packages/google/cloud/iterator.py", line 155, in next
    result = self._item_to_value(self._parent, item)
  File "/home/dev-sean/BQTesting/local/lib/python2.7/site-packages/google/cloud/bigquery/client.py", line 382, in _item_to_job
    return iterator.client.job_from_resource(resource)
  File "/home/dev-sean/BQTesting/local/lib/python2.7/site-packages/google/cloud/bigquery/client.py", line 165, in job_from_resource
    return CopyJob.from_api_repr(resource, self)
  File "/home/dev-sean/BQTesting/local/lib/python2.7/site-packages/google/cloud/bigquery/job.py", line 745, in from_api_repr
    for source_config in config['sourceTables']:
KeyError: 'sourceTables'

5. Steps to reproduce

• instantiate the list_jobs iterator with parameter all_users=True
• iterate through the jobs
• as it iterates the error is produced when converting response from_api_repr

6. Code example

jobs = bqClient.list_jobs(all_users=True)
for i,job in enumerate(jobs):
    print job.name

[tseaver]

@sean185 Thanks for the report! Based on the traceback, the job being constructed is a "copy" job: does that seem plausible to you?

One thing you could do to help identify the problem would be to set a breakpoint at the failure, and report the contents of the resource dict. E.g.:

jobs = bqClient.list_jobs(all_users=True)
try:
    jobs = list(jobs)
except KeyError:
    import pdb; pdb.set_trace()

and then "step down" into the traceback until you get to the bottom, and pp resource.

[sean185]

@tseaver Thank you for responding so quickly!

Yes I think it is possible that Copy Jobs are not converted properly, since the list_jobs iterator fails when it comes listing out the copy job processed that day.

Apologies, I am not too familiar with the use of pdb, but just using your code and typing s to step through, I got the following:

-> import pdb; pdb.set_trace()
(Pdb) s
--Return--
> /home/dev-sean/BQTesting/pybq_test.py(76)<module>()->None
-> main()
(Pdb) s
Exception AttributeError: "'NoneType' object has no attribute 'path'" in <function _remove at 0x7ff60004f6e0> ignored

and so before I could type pp resource, the program has already exited.

[tseaver]

@sean185 I'm guessing that the copy job which causes this failure was created with a single sourceTable, rather than a sequence of sourceTables (it would have been created with a different client, as we don't expose that option).

[sean185]

@tseaver Man, this is amazing, I think this means it'll be fixed in the next patch or something? You guys are awesome! For now though I'll have to stick with not being able to list all jobs at all =x

So when should I close this issue?

[tseaver]

@sean185

I think this means it'll be fixed in the next patch or something? You guys are awesome! For now though I'll have to stick with not being able to list all jobs at all.

I don't think there is any way to filter / narrow the set of jobs returned by the API. You could try applying the patch from the PR to your local copy.

So when should I close this issue?

It will be closed automatically when PR #2884 gets merged.