paging vs infinite API's - Cloud Storage, but more broadly applicable...

@anthmgoogle

In looking at the cloud storage API's, they are really a mapping of our internal protobuf apis to a client surface rather than going with what we had hoped to be the more natural view.

For example we show the following for list:

 String bucketName = "my_unique_bucket";
 String directory = "my_directory/";
 Page<Blob> blobs = storage.list(bucketName, BlobListOption.currentDirectory(),
     BlobListOption.prefix(directory));
 Iterator<Blob> blobIterator = blobs.iterateAll();
 while (blobIterator.hasNext()) {
   Blob blob = blobIterator.next();
   // do something with the blob
 }

In actuality the usage is more like:

 String bucketName = "my_unique_bucket";
 String directory = "my_directory/";
 Page<Blob> blobs = storage.list(bucketName, BlobListOption.currentDirectory(),
     BlobListOption.prefix(directory));
 do {
    Iterator<Blob> blobIterator = blobs.iterateAll();
    while (blobIterator.hasNext()) {
      Blob blob = blobIterator.next();
      // do something with the blob
    }
    blobs = blobs.getNextPage();
  } while ( blobs != null);

In a sense there are cases where it makes sense to use paging, such as doing so in a web app, but there are also many cases where we should hide our complexity from our users and just provide a single blobs iterator.

I don't wish to block GA with this as I don't think it makes sense, but this is something we should fix in the autogenerated libraries. Developers should be able to choose between a paged api, for webapps and the like, and a continuous api (that we can optimize behind the scenes).

@anthmgoogle FYI
@waprin FYI
@jabubake FYI
@garrettjonesgoogle
@jgeewax FYI

@michaelbausor

@michaelbausor is actually working on the paging interface right now (#1568). Mike, could you chime in here?

@lesv

@lesv I am not sure I fully understand. The iterateAll() method does handle paging behind the scenes and iterate over all elements. So in your second code block, the calls to getNextPage() are not required - blobs.iterateAll() will make those calls in the background and iterate over everything.

This is the same in the autogenerated paging implementation, which provides an iterateAllElements() method in the response object. I am currently working to unify the autogenerated and manual paging implementations, but we will definitely keep the ability to iterate all results in a single call, with paging handled automatically.

GCJ paging impl: https://github.com/GoogleCloudPlatform/google-cloud-java/blob/master/google-cloud-core/src/main/java/com/google/cloud/PageImpl.java#L91
GAX paged response impl: https://github.com/googleapis/gax-java/blob/master/src/main/java/com/google/api/gax/grpc/PagedListResponseImpl.java#L62

Cool - it's not the first time that I've missed something, probably won't be the last. :(

Since I have everyone's attention, should we change our example to be:

 String bucketName = "my_unique_bucket";
 String directory = "my_directory/";
 Page<Blob> blobs = storage.list(bucketName, BlobListOption.currentDirectory(),
     BlobListOption.prefix(directory));
 for (Blob blob : blobs.iterateAll()) {

   // do something with the blob
 }

@lesv

@lesv As part of the update to the paging interface (#1894 have updated our samples to use the iteration style you suggest above. For example here: 5f5a859#diff-d0f13847f2d8f6c8b2a3b1bd238814feR115