Monitoring system tests: Collision on group.create

From: https://travis-ci.org/GoogleCloudPlatform/gcloud-python/builds/151551907#L1081-L1091

======================================================================
ERROR: test_list_group_members (monitoring.TestMonitoringGroups)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/travis/build/GoogleCloudPlatform/gcloud-python/system_tests/monitoring.py", line 263, in test_list_group_members
    group.create()
  File "/home/travis/build/GoogleCloudPlatform/gcloud-python/.tox/system-tests3/lib/python3.4/site-packages/gcloud/monitoring/group.py", line 178, in create
    method='POST', path=path, data=self._to_dict())
  File "/home/travis/build/GoogleCloudPlatform/gcloud-python/.tox/system-tests3/lib/python3.4/site-packages/gcloud/connection.py", line 347, in api_request
    error_info=method + ' ' + url)
gcloud.exceptions.ServiceUnavailable: 503 Write collision, please retry. (POST https://monitoring.googleapis.com/v3/projects/precise-truck-742/groups/)

@rimey

@rimey Can you weigh in here?

Smells like we just need to add a RetryErrors(ServiceUnavailable) here.

I think #2095 addresses this error.

@dhermes

@dhermes I'm having trouble assigning this issue to @supriyagarg. Does she need to be a member of GoogleCloudPlatform?

I think she needs to be a member of the GoogleCloudPlatform "Contributors" team.

See my comment on #2092. This error is slightly surprising. Let's find out more about in exactly what circumstances it occurs, and whether it is deterministic.

Closed by #2095

@tseaver

@tseaver How hard is it to reproduce these errors? I tried on my local machine and trying on Compute Engine right now and haven't been able to repro. I am doing this because I want to see if there is an equivalent to error['reason'] == 'rateLimitExceeded' in the caught error

@dhermes

@dhermes I saw them multiple times the day I was trying to squash all the flaky issues: if I had to guess, I would say that the issue would be related to load on the back-end.

@tseaver

I don't know why I never tried before, but just running nosetests system_tests/monitoring.py --pdb -vv in two separate bash shells at the same time forced a 503 on the first try.

FWIW @tseaver the 503 occurred in a 404-wrapped delete during tearDown of TestMonitoringGroups.test_group_hierarchy.

From the debugger:

(Pdb) print(json.dumps(response, indent=2))
{
  "status": "503",
  "alternate-protocol": "443:quic",
  "content-length": "247",
  "x-xss-protection": "1; mode=block",
  "x-content-type-options": "nosniff",
  "transfer-encoding": "chunked",
  "vary": "Origin, X-Origin, Referer",
  "server": "ESF",
  "-content-encoding": "gzip",
  "cache-control": "private",
  "date": "Mon, 22 Aug 2016 14:38:43 GMT",
  "x-frame-options": "SAMEORIGIN",
  "alt-svc": "quic=\":443\"; ma=2592000; v=\"35,34,33,32,31,30\"",
  "content-type": "application/json; charset=UTF-8"
}
(Pdb) print(content)
{
  "error": {
    "code": 503,
    "message": "Write collision, please retry.",
    "status": "UNAVAILABLE",
    "details": [
      {
        "@type": "type.googleapis.com/google.rpc.RetryInfo",
        "retryDelay": "1.000s"
      }
    ]
  }
}

Also note that

(Pdb) exc = make_exception(response, content, error_info=method + ' ' + url)
(Pdb) exc.errors
[]
(Pdb) exc._errors
()

because the error payload doesn't have the key payload['error']['errors'].

What we really want is a check like payload['error']['status'] == u'UNAVAILABLE' just like in system_tests.bigquery._rate_limit_exceeded.

feat: support resource_tags for table (#2093)

refactor: reorganize the sqlglot scalar compiler layout - part 2 (#2093)

Navigation Menu

Repository navigation

Monitoring system tests: Collision on group.create #2093

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Monitoring system tests: Collision on group.create #2093

Description

Activity

dhermes commented on Aug 11, 2016

tseaver commented on Aug 11, 2016

tseaver commented on Aug 11, 2016

rimey commented on Aug 11, 2016

rimey commented on Aug 11, 2016

rimey commented on Aug 12, 2016

dhermes commented on Aug 18, 2016

dhermes commented on Aug 18, 2016

tseaver commented on Aug 19, 2016

dhermes commented on Aug 22, 2016

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions