Basic polling helper function for long running operations/jobs

Several libraries with long running operation are lacking basic helper functions for polling

Considering that implementing exponential backoff properly is a 15 line affair which clutters up code it might be good to have a util function that properly implements exponential backoff and then allow things to call it, like in bigquery.job.Jobs

def wait(self):
    utils.exponential_backoff(self.begin, self.reload, self.status)

I've also suggested this for things like long-running operations

@elibixby

@elibixby We have a utility like that now for system testing. We could move it (and maybe the error-based and result-based ones) into the library itself, which might look like:

from gcloud.retry import RetryInstanceState

def _job_done(job):
    return job.state.lower() == 'done'

until_done = RetryInstanceState(_job_done)

class _BaseJob(object):
    ...
    @until_done
    def wait(self):
        self.reload()

@dhermes We need to factor out the google.longrunning.operations bit, including making the name pattern and metadata lookup bits less smelly. If we did, then we could add an Operation.wait method.

@tseaver

@tseaver Good point. We could even return an Operation instance from async uploads for BigQuery and have the concept map across packages.

@bjwatson

@bjwatson Another grpc / GAX request: can you release wrappers for google.longrunning? We have a hacky self-generated version, but would be much better off using one "from the horse's mouth."

Note that this is just the gRPC bits: we are already using the "bare" proto-generated stuff from googleapis_common_protos.

@tseaver

@tseaver That's currently a WIP for us at the GAX level. @garrettjonesgoogle is leading this effort.

If you just need the GRPC generated code, then I think we can do that for you quickly.

@bjwatson

@bjwatson We can use the stub-based (gRPC stubs) approach without GAX for now: basically, all we need is a way to call GetOperation for polling purposes. Later on, it would be good if DeleteOperation and CancelOperation were possible (I don't think we have a usecase for ListOperations).

@tseaver

@tseaver I tried doing that today using an internal build server we setup to introduce more controllability into our release process (previously done on dev workstations with varying environments). Unfortunately, I hit a few issues. I'll try again tomorrow with help from @ethanbao and @geigerj.

gRPC longrunning package available here: https://pypi.python.org/pypi/grpc-google-longrunning-v2/

I think the VKit team is still working out the full story for "meta-APIs" like longrunning, so I'll hold off the GAX package if you guys don't need it now.

@geigerj

@geigerj Thanks! I'll work on folding it in. One thing I note is that the 0.8.1 release doesn't have the v2 in the path: is that intentional?

@tseaver

@tseaver The protos are unversioned, and the lack of version in the path is a direct consequence.

@geigerj

@geigerj OK, WFM.

@jonparrott

I've already released 1.3.x of https://pypi.python.org/pypi/googleapis-common-protos with grpcio as a install_requires. @jonparrott if I change this to an extras_require, does that satisfy the needs of our GAE Standard clients?

The alternative is extra engineering work to separate the longrunning messages and services into two separate packages, which is a bit of a fragile post-processing hack we would have to do to the gRPC output (unless they added a feature to explicitly support this mode).

@jmuk

FYI @jmuk. I know you've had similar conversations on the Ruby side.

@bjwatson

@bjwatson that could work, this library would have to update the grpc extra to require googleapi-common-protos[grpc]. I'm not 100% sure this will work, so it probably needs to be tested before we go and publish the packages.

(Note: users are already having fun with grpc).

btw testpypi exists for the purposes of testing: https://testpypi.python.org/pypi

@jonparrott

thanks @jonparrott. we'll try this out with testpypi

@tseaver

@tseaver @jonparrott The updated package with an optional [grpc] dependency is available at https://testpypi.python.org/pypi/googleapis-common-protos (version 1.3.2).

pip install googleapis-common-protos
-or-
pip install googleapis-common-protos[grpc]

PLMK if this looks good, so I can publish it to the regular pypi.

FYI, the test package has long since been published on regular PyPI. We are currently at 1.3.4.

This exists in gax now, I'm gonna close this.

Apologies for resurrecting this thread.

I am trying to implement longrunning operation interface for one of my APIs. I tried installing apis-common-protos using googleapis-common-protos and googleapis-common-protos[grpc]. The operations_grpc_pb2.py from installed package does contain client stub, but no servicer interface is available.

Am I missing anything here or are there any specific instructions to generate the same?

@bjwatson

I am bit stuck on this. @bjwatson @theacodes any pointers would be really appreciated.

chore(main): release 3.29.0 (#2117)

refactor: support and, or, xor for sqlglot compiler (#2117)

Navigation Menu

Repository navigation

Basic polling helper function for long running operations/jobs #2117

25 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Basic polling helper function for long running operations/jobs #2117

Description

Activity

dhermes commented on Aug 18, 2016

tseaver commented on Aug 18, 2016

dhermes commented on Aug 18, 2016

tseaver commented on Aug 18, 2016

tseaver commented on Aug 18, 2016

bjwatson commented on Aug 18, 2016

tseaver commented on Aug 18, 2016

bjwatson commented on Aug 19, 2016

geigerj commented on Aug 25, 2016

tseaver commented on Aug 25, 2016

geigerj commented on Aug 25, 2016

tseaver commented on Aug 25, 2016

25 remaining items

bjwatson commented on Sep 2, 2016

bjwatson commented on Sep 2, 2016

theacodes commented on Sep 2, 2016

theacodes commented on Sep 2, 2016

bjwatson commented on Sep 2, 2016

bjwatson commented on Sep 2, 2016

bjwatson commented on Sep 12, 2016

theacodes commented on Jun 27, 2017

asandeep commented on Oct 14, 2019

asandeep commented on Oct 21, 2019

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions