people-and-planet-ai.image-classification.e2e_test: test_predict failed

[flaky-bot]

Note: #6157 was also for this test, but it was closed more than 10 days ago. So, I didn't mark it flaky.

---

commit: 3b5a4bb
buildURL: Build Status, Sponge
status: failed

Test output

Traceback (most recent call last):
  File "/workspace/people-and-planet-ai/image-classification/e2e_test.py", line 112, in model_endpoint_id
    PROJECT, REGION, MODEL_PATH, MODEL_ENDPOINT, endpoint_id
  File "/workspace/people-and-planet-ai/image-classification/deploy_model.py", line 80, in deploy_model
    deployed_model = response.result()
  File "/workspace/people-and-planet-ai/image-classification/.nox/py-3-7/lib/python3.7/site-packages/google/api_core/future/polling.py", line 135, in result
    raise self._exception
google.api_core.exceptions.InternalServerError: 500 INTERNAL

[flaky-bot]

Looks like this issue is flaky. 😟

I'm going to leave this open and stop commenting.

A human should fix and close this.

---

When run at the same commit (3b5a4bb), this test passed in one build (Build Status, Sponge) and failed in another build (Build Status, Sponge).

[dandhlee]

Oh no it's back D:

[dandhlee]

Closing this for now, this is the same issue we've had from the last time this test was flaky, hoping that it's a one time flake. If it happens again I'll reach back to the product team.

[flaky-bot]

Oops! Looks like this issue is still flaky. It failed again. 😬

I reopened the issue, but a human will need to close it again.

---

commit: 73fa296
buildURL: Build Status, Sponge
status: failed

Test output

Traceback (most recent call last):
  File "/workspace/people-and-planet-ai/image-classification/.nox/py-3-6/lib/python3.6/site-packages/google/api_core/grpc_helpers.py", line 67, in error_remapped_callable
    return callable_(*args, **kwargs)
  File "/workspace/people-and-planet-ai/image-classification/.nox/py-3-6/lib/python3.6/site-packages/grpc/_channel.py", line 946, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/workspace/people-and-planet-ai/image-classification/.nox/py-3-6/lib/python3.6/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
    raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.NOT_FOUND
	details = "Model `projects/python-docs-samples-tests/locations/us-central1/models/1590773423066316800` is not found."
	debug_error_string = "{"created":"@1627637689.682554879","description":"Error received from peer ipv4:74.125.142.95:443","file":"src/core/lib/surface/call.cc","file_line":1069,"grpc_message":"Model `projects/python-docs-samples-tests/locations/us-central1/models/1590773423066316800` is not found.","grpc_status":5}"
>
The above exception was the direct cause of the following exception:
Traceback (most recent call last):

File "/workspace/people-and-planet-ai/image-classification/e2e_test.py", line 112, in model_endpoint_id

PROJECT, REGION, MODEL_PATH, MODEL_ENDPOINT, endpoint_id

File "/workspace/people-and-planet-ai/image-classification/deploy_model.py", line 77, in deploy_model

traffic_split={"0": 100},

File "/workspace/people-and-planet-ai/image-classification/.nox/py-3-6/lib/python3.6/site-packages/google/cloud/aiplatform_v1/services/endpoint_service/client.py", line 905, in deploy_model

response = rpc(request, retry=retry, timeout=timeout, metadata=metadata,)

File "/workspace/people-and-planet-ai/image-classification/.nox/py-3-6/lib/python3.6/site-packages/google/api_core/gapic_v1/method.py", line 145, in call

return wrapped_func(*args, **kwargs)

File "/workspace/people-and-planet-ai/image-classification/.nox/py-3-6/lib/python3.6/site-packages/google/api_core/grpc_helpers.py", line 69, in error_remapped_callable

six.raise_from(exceptions.from_grpc_error(exc), exc)

File "", line 3, in raise_from

google.api_core.exceptions.NotFound: 404 Model projects/python-docs-samples-tests/locations/us-central1/models/1590773423066316800 is not found.

[parthea]

This error google.api_core.exceptions.NotFound: 404 Model is different than the OP, which is an internal 500 error. I'm going to close this issue and see if it comes back.

[flaky-bot]

Oops! Looks like this issue is still flaky. It failed again. 😬

I reopened the issue, but a human will need to close it again.

---

commit: 73fa296
buildURL: Build Status, Sponge
status: failed

Test output

Traceback (most recent call last):
  File "/workspace/people-and-planet-ai/image-classification/.nox/py-3-8/lib/python3.8/site-packages/google/api_core/grpc_helpers.py", line 67, in error_remapped_callable
    return callable_(*args, **kwargs)
  File "/workspace/people-and-planet-ai/image-classification/.nox/py-3-8/lib/python3.8/site-packages/grpc/_channel.py", line 946, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/workspace/people-and-planet-ai/image-classification/.nox/py-3-8/lib/python3.8/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
    raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.NOT_FOUND
	details = "Model `projects/python-docs-samples-tests/locations/us-central1/models/1590773423066316800` is not found."
	debug_error_string = "{"created":"@1627645645.653339393","description":"Error received from peer ipv4:74.125.142.95:443","file":"src/core/lib/surface/call.cc","file_line":1069,"grpc_message":"Model `projects/python-docs-samples-tests/locations/us-central1/models/1590773423066316800` is not found.","grpc_status":5}"
>
The above exception was the direct cause of the following exception:
Traceback (most recent call last):

File "/workspace/people-and-planet-ai/image-classification/e2e_test.py", line 111, in model_endpoint_id

deployed_model_id = deploy_model.deploy_model(

File "/workspace/people-and-planet-ai/image-classification/deploy_model.py", line 65, in deploy_model

response = client.deploy_model(

File "/workspace/people-and-planet-ai/image-classification/.nox/py-3-8/lib/python3.8/site-packages/google/cloud/aiplatform_v1/services/endpoint_service/client.py", line 905, in deploy_model

response = rpc(request, retry=retry, timeout=timeout, metadata=metadata,)

File "/workspace/people-and-planet-ai/image-classification/.nox/py-3-8/lib/python3.8/site-packages/google/api_core/gapic_v1/method.py", line 145, in call

return wrapped_func(*args, **kwargs)

File "/workspace/people-and-planet-ai/image-classification/.nox/py-3-8/lib/python3.8/site-packages/google/api_core/grpc_helpers.py", line 69, in error_remapped_callable

six.raise_from(exceptions.from_grpc_error(exc), exc)

File "", line 3, in raise_from

google.api_core.exceptions.NotFound: 404 Model projects/python-docs-samples-tests/locations/us-central1/models/1590773423066316800 is not found.