Skip to content

spanner.cloud-client.backup_sample_test: test_update_backup failed #3241

Closed
#3688
#3468
Closed
spanner.cloud-client.backup_sample_test: test_update_backup failed#3241
#3688
#3468
Assignees
tmatsuo
Labels
api: spannerIssues related to the Spanner API.priority: p2Moderately-important priority. Fix may not be included in next release.type: bugError or flaw in code with unintended results or allowing sub-optimal usage patterns.

Description

@flaky-bot
flaky-bot
bot
opened on Mar 31, 2020

This test failed!

To configure my behavior, see the Build Cop Bot documentation.

If I'm commenting on this issue too often, add the buildcop: quiet label and
I will stop commenting.

commit: 7bc4783
buildURL: Build Status, Sponge
status: failed

Activity

  1. flaky-bot
    added
    buildcop: issue
    priority: p1Important issue which blocks shipping the next release. Will be fixed prior to next release.
    type: bugError or flaw in code with unintended results or allowing sub-optimal usage patterns.
    on Mar 31, 2020
  2. blunderbuss-gcf
    assigned
    leahecole
    on Mar 31, 2020

  3. flaky-bot commented on Apr 1, 2020

    @flaky-bot
    flaky-botBot
    on Apr 1, 2020
    Author

    Test passed for commit 9e64bff (Build Status, Sponge)! Closing this issue.

  4. flaky-bot
    closed this as completedon Apr 1, 2020
  5. flaky-bot
    reopened this on Apr 1, 2020
  6. flaky-bot
    added
    buildcop: flaky
    on Apr 1, 2020

  7. flaky-bot commented on Apr 1, 2020

    @flaky-bot
    flaky-botBot
    on Apr 1, 2020
    Author

    Looks like this issue is flaky. 😟

    I'm going to leave this open and stop commenting.

    A human should fix and close this.

    commit: e943bff
    buildURL: Build Status, Sponge
    status: failed

  8. leahecole
    assigned
    mmmarklu
    and unassigned
    leahecole
    on Apr 1, 2020
  9. JustinBeckwith
    added
    api: spannerIssues related to the Spanner API.
    on Apr 2, 2020
  10. yoshi-automation
    added
    🚨This issue needs some love.
    on Apr 17, 2020
  11. tmatsuo
    closed this as completedon Apr 21, 2020
  12. flaky-bot
    reopened this on Apr 22, 2020

  13. 10 remaining items

  14. tmatsuo commented on Apr 27, 2020

    @tmatsuo
    tmatsuo
    on Apr 27, 2020
    Contributor

    It seems like the default timeout for create_backup is 30 seconds.

    We can add timeout parameter to create_backup call for mitigating the test failure, but it is also too long for the server to return a Long Running Operation. If the server is expected to take more than 30 seconds frequently enough (currently it seems like so), we should change the client library's default value.

    @larkee WDYT?

    Update: There is a new config for the timeout. I filed googleapis/python-spanner#68

  15. tmatsuo
    assigned
    tmatsuo
    and unassigned
    larkee
    on Apr 27, 2020
  16. tmatsuo
    added
    priority: p2Moderately-important priority. Fix may not be included in next release.
    and removed
    priority: p1Important issue which blocks shipping the next release. Will be fixed prior to next release.
    on Apr 27, 2020

  17. tmatsuo commented on Apr 27, 2020

    @tmatsuo
    tmatsuo
    on Apr 27, 2020
    Contributor

    Blocked by googleapis/python-spanner#68

  18. yoshi-automation
    removed
    🚨This issue needs some love.
    on Apr 27, 2020
  19. tmatsuo
    closed this as completedin #3688on May 5, 2020
  20. flaky-bot
    reopened this on May 6, 2020

  21. flaky-bot commented on May 6, 2020

    @flaky-bot
    flaky-botBot
    on May 6, 2020
    Author

    Oops! Looks like this issue is still flaky. It failed again. 😬

    I reopened the issue, but a human will need to close it again.

    commit: 32777eb
    buildURL: Build Status, Sponge
    status: failed

    Test output
    Traceback (most recent call last):
  File "/tmpfs/src/github/python-docs-samples/spanner/cloud-client/.nox/py-3-8/lib/python3.8/site-packages/google/api_core/grpc_helpers.py", line 57, in error_remapped_callable
    return callable_(*args, **kwargs)
  File "/tmpfs/src/github/python-docs-samples/spanner/cloud-client/.nox/py-3-8/lib/python3.8/site-packages/grpc/_channel.py", line 826, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/tmpfs/src/github/python-docs-samples/spanner/cloud-client/.nox/py-3-8/lib/python3.8/site-packages/grpc/_channel.py", line 729, in _end_unary_response_blocking
    raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.DEADLINE_EXCEEDED
	details = "Deadline Exceeded"
	debug_error_string = "{"created":"@1588771706.167677943","description":"Error received from peer ipv4:108.177.112.95:443","file":"src/core/lib/surface/call.cc","file_line":1056,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>

    The above exception was the direct cause of the following exception:
    

    Traceback (most recent call last):
    
File "/tmpfs/src/github/python-docs-samples/spanner/cloud-client/backup_sample_test.py", line 97, in test_update_backup
    
backup_sample.update_backup(INSTANCE_ID, BACKUP_ID)
    
File "/tmpfs/src/github/python-docs-samples/spanner/cloud-client/backup_sample.py", line 232, in update_backup
    
backup.update_expire_time(new_expire_time)
    
File "/tmpfs/src/github/python-docs-samples/spanner/cloud-client/.nox/py-3-8/lib/python3.8/site-packages/google/cloud/spanner_v1/backup.py", line 249, in update_expire_time
    
api.update_backup(backup_update, update_mask, metadata=metadata)
    
File "/tmpfs/src/github/python-docs-samples/spanner/cloud-client/.nox/py-3-8/lib/python3.8/site-packages/google/cloud/spanner_admin_database_v1/gapic/database_admin_client.py", line 1192, in update_backup
    
return self._inner_api_calls["update_backup"](
    
File "/tmpfs/src/github/python-docs-samples/spanner/cloud-client/.nox/py-3-8/lib/python3.8/site-packages/google/api_core/gapic_v1/method.py", line 143, in call
    
return wrapped_func(*args, **kwargs)
    
File "/tmpfs/src/github/python-docs-samples/spanner/cloud-client/.nox/py-3-8/lib/python3.8/site-packages/google/api_core/retry.py", line 281, in retry_wrapped_func
    
return retry_target(
    
File "/tmpfs/src/github/python-docs-samples/spanner/cloud-client/.nox/py-3-8/lib/python3.8/site-packages/google/api_core/retry.py", line 184, in retry_target
    
return target()
    
File "/tmpfs/src/github/python-docs-samples/spanner/cloud-client/.nox/py-3-8/lib/python3.8/site-packages/google/api_core/timeout.py", line 214, in func_with_timeout
    
return func(*args, **kwargs)
    
File "/tmpfs/src/github/python-docs-samples/spanner/cloud-client/.nox/py-3-8/lib/python3.8/site-packages/google/api_core/grpc_helpers.py", line 59, in error_remapped_callable
    
six.raise_from(exceptions.from_grpc_error(exc), exc)
    
File "", line 3, in raise_from
    
google.api_core.exceptions.DeadlineExceeded: 504 Deadline Exceeded

  22. tmatsuo commented on May 6, 2020

    @tmatsuo
    tmatsuo
    on May 6, 2020
    Contributor

    The last build failure is:

    Traceback (most recent call last):
  File "/tmpfs/src/github/python-docs-samples/spanner/cloud-client/.nox/py-3-8/lib/python3.8/site-packages/google/api_core/grpc_helpers.py", line 57, in error_remapped_callable
    return callable_(*args, **kwargs)
  File "/tmpfs/src/github/python-docs-samples/spanner/cloud-client/.nox/py-3-8/lib/python3.8/site-packages/grpc/_channel.py", line 826, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/tmpfs/src/github/python-docs-samples/spanner/cloud-client/.nox/py-3-8/lib/python3.8/site-packages/grpc/_channel.py", line 729, in _end_unary_response_blocking
    raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.DEADLINE_EXCEEDED
	details = "Deadline Exceeded"
	debug_error_string = "{"created":"@1588771706.167677943","description":"Error received from peer ipv4:108.177.112.95:443","file":"src/core/lib/surface/call.cc","file_line":1056,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/tmpfs/src/github/python-docs-samples/spanner/cloud-client/backup_sample_test.py", line 97, in test_update_backup
    backup_sample.update_backup(INSTANCE_ID, BACKUP_ID)
  File "/tmpfs/src/github/python-docs-samples/spanner/cloud-client/backup_sample.py", line 232, in update_backup
    backup.update_expire_time(new_expire_time)
  File "/tmpfs/src/github/python-docs-samples/spanner/cloud-client/.nox/py-3-8/lib/python3.8/site-packages/google/cloud/spanner_v1/backup.py", line 249, in update_expire_time
    api.update_backup(backup_update, update_mask, metadata=metadata)
  File "/tmpfs/src/github/python-docs-samples/spanner/cloud-client/.nox/py-3-8/lib/python3.8/site-packages/google/cloud/spanner_admin_database_v1/gapic/database_admin_client.py", line 1192, in update_backup
    return self._inner_api_calls["update_backup"](
  File "/tmpfs/src/github/python-docs-samples/spanner/cloud-client/.nox/py-3-8/lib/python3.8/site-packages/google/api_core/gapic_v1/method.py", line 143, in __call__
    return wrapped_func(*args, **kwargs)
  File "/tmpfs/src/github/python-docs-samples/spanner/cloud-client/.nox/py-3-8/lib/python3.8/site-packages/google/api_core/retry.py", line 281, in retry_wrapped_func
    return retry_target(
  File "/tmpfs/src/github/python-docs-samples/spanner/cloud-client/.nox/py-3-8/lib/python3.8/site-packages/google/api_core/retry.py", line 184, in retry_target
    return target()
  File "/tmpfs/src/github/python-docs-samples/spanner/cloud-client/.nox/py-3-8/lib/python3.8/site-packages/google/api_core/timeout.py", line 214, in func_with_timeout
    return func(*args, **kwargs)
  File "/tmpfs/src/github/python-docs-samples/spanner/cloud-client/.nox/py-3-8/lib/python3.8/site-packages/google/api_core/grpc_helpers.py", line 59, in error_remapped_callable
    six.raise_from(exceptions.from_grpc_error(exc), exc)
  File "<string>", line 3, in raise_from
google.api_core.exceptions.DeadlineExceeded: 504 Deadline Exceeded

  23. tmatsuo commented on May 6, 2020

    @tmatsuo
    tmatsuo
    on May 6, 2020
    Contributor

    The current deadline is 30 seconds for UpdateBackup.

    I'll talk with the Spanner folks to decide how to handle this case.

  24. tmatsuo commented on May 7, 2020

    @tmatsuo
    tmatsuo
    on May 7, 2020
    Contributor

    The spanner team is going to bump the deadline, so we don't have any action items other than wait for the next release.

  25. tmatsuo commented on May 27, 2020

    @tmatsuo
    tmatsuo
    on May 27, 2020
    Contributor

    Got the new release, closing

  26. tmatsuo
    closed this as completedon May 27, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

api: spannerIssues related to the Spanner API.priority: p2Moderately-important priority. Fix may not be included in next release.type: bugError or flaw in code with unintended results or allowing sub-optimal usage patterns.

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions