Repository navigation
AttributionError raised When Google Ads Server Error Occurred #542
Description
Describe the bug:
AttributeError: 'NoneType' object has no attribute 'Call'
Stacktrace
status = StatusCode.INTERNAL
details = "Internal error encountered....
grpc._channel._SingleThreadedRendezvous: <_SingleThreadedRendezvous of RPC that terminated with:
status = StatusCode.INTERNAL
details = "Internal error encountered."
debug_error_string = " "created": "@1638903791.733258205",
"description": "Error received from peer ipv4:216.58.197.202:443",
"file": "src/core/lib/surface/call.cc",
"file_line": 1069,
"grpc_message": "Internal error encountered.",
"grpc_status": 13
}
"
debug_error_string = "{"created":"@1638903791.733258205","description":"Error received from peer ipv4:216.58.197.202:443","file":"src/core/lib/surface/call.cc","file_line":1069,"grpc_message":"Internal error encountered.","grpc_status":13}"
File "/var/task/google/ads/googleads/v9/services/services/google_ads_service/client.py", line 3282, in search_stream
response = rpc(
└ <google.api_core.gapic_v1.method._GapicCallable object at 0x7fb798f9a430>
File "/var/task/google/api_core/gapic_v1/method.py", line 154, in __call__
return wrapped_func(*args, **kwargs)
│ │ └ {'metadata': [('x-goog-request-params', 'customer_id=**MASKING**'), ('x-goog-api-client', 'gl-python/3.8.11 grpc/1.41.1 gax/2....
│ └ (customer_id: "**MASKING**"
│ query: "SELECT \n customer.id,\n customer.descriptive_name,\n cus...
└ <function _wrap_stream_errors.<locals>.error_remapped_callable at 0x7fb798f9eaf0>
File "/var/task/google/api_core/grpc_helpers.py", line 160, in error_remapped_callable
raise exceptions.from_grpc_error(exc) from exc
│ └ <function from_grpc_error at 0x7fb79d25a700>
└ <module 'google.api_core.exceptions' from '/var/task/google/api_core/exceptions.py'>
File "/var/task/google/api_core/exceptions.py", line 532, in from_grpc_error
if isinstance(rpc_exc, grpc.Call) or _is_informative_grpc_error(rpc_exc):
│ │ │ └ <_SingleThreadedRendezvous of RPC that terminated with:
│ │ │ status = StatusCode.INTERNAL
│ │ │ details = "Internal error encountered....
│ │ └ <function _is_informative_grpc_error at 0x7fb79d25a5e0>
│ └ None
└ <_SingleThreadedRendezvous of RPC that terminated with:
status = StatusCode.INTERNAL
details = "Internal error encountered....
AttributeError: 'NoneType' object has no attribute 'Call'
(sorry for formatting)
Steps to Reproduce:
it is not easy to reproduce because it needs server error to reproduce the bug.
And as a rule of thumb, not all server errors raise AttributeError. In some cases, ServerError exception is normally raised.
Expected behavior:
ServerError exception raised
Client library version and API version:
Client library version: 14.1.0 (latest)
Google Ads API version: v9
aiohttp @ file:///Users/sym/Library/Caches/pypoetry/artifacts/05/07/33/33717ed661d3ff4fefae447a53297894172e5164474c91063085d43330/aiohttp-3.8.0-cp38-cp38-macosx_10_9_x86_64.whl
aiosignal @ file:///Users/sym/Library/Caches/pypoetry/artifacts/4c/98/1b/ee90d29611d2aaeb4a2b40c1509ca578cccf552d05a53563f4f8bf9bb3/aiosignal-1.2.0-py3-none-any.whl
appdirs @ file:///Users/sym/Library/Caches/pypoetry/artifacts/68/c7/33/ef0f6af8d1aa906f1a5641762fe5d6f47d565c5a4b3482f1c6fb0f5f20/appdirs-1.4.4-py2.py3-none-any.whl
argcomplete @ file:///Users/sym/Library/Caches/pypoetry/artifacts/96/70/57/5875e2d640c712de1319caa20ce7168ed5a211a3c87554b04e123b347b/argcomplete-1.12.3-py2.py3-none-any.whl
arrow @ file:///Users/sym/Library/Caches/pypoetry/artifacts/27/b2/c5/96652e26f28b571e8c677545083f3d8e295914b90344709c00f7ad2ca8/arrow-1.1.1-py3-none-any.whl
async-timeout @ file:///Users/sym/Library/Caches/pypoetry/artifacts/84/cd/e4/c9346fd7aaa2b90be31afd8151ca9fc849baf4f1dc93714e23202c2a92/async_timeout-4.0.0-py3-none-any.whl
attrs @ file:///Users/sym/Library/Caches/pypoetry/artifacts/1a/aa/39/10d6d07084f186f8cf6963cb033440402ad5088bb94d712239170f2ef6/attrs-21.2.0-py2.py3-none-any.whl
aws-embedded-metrics @ file:///Users/sym/Library/Caches/pypoetry/artifacts/cc/61/7c/f721bc7db7a765043ccf5491a0b0af4b2aa8d843e2d260cd0746104fa4/aws_embedded_metrics-1.0.7-py3-none-any.whl
backports.entry-points-selectable @ file:///Users/sym/Library/Caches/pypoetry/artifacts/87/38/bc/010a813c45cba13501d5134886066f1a3dfe52f3ae0fe28996a8035dcf/backports.entry_points_selectable-1.1.1-py2.py3-none-any.whl
black @ file:///Users/sym/Library/Caches/pypoetry/artifacts/38/f4/61/550fd050005dc1c64850374a7f98dd941a4633d72e87f147bebf26bdc3/black-20.8b1.tar.gz
boto3 @ file:///Users/sym/Library/Caches/pypoetry/artifacts/ae/4a/f7/bec32731294b3f6bc791e2947b1015d0cb28c7d4cd505e65c812abbc9a/boto3-1.16.63-py2.py3-none-any.whl
botocore @ file:///Users/sym/Library/Caches/pypoetry/artifacts/58/6e/f5/be863909d3ea80cc6665ccf7b99324ab0d8a08681ecea2815149f35fd2/botocore-1.19.63-py2.py3-none-any.whl
cachetools @ file:///Users/sym/Library/Caches/pypoetry/artifacts/ca/68/fb/a94522dd8c8d081803f16d6ec74ce9a678a827247431e2a8339f25506f/cachetools-4.2.4-py3-none-any.whl
certifi @ file:///Users/sym/Library/Caches/pypoetry/artifacts/77/74/3e/daf57610ff13227e3b3f50a78487055f6ac94cc9363abd2463be34c5e9/certifi-2021.10.8-py2.py3-none-any.whl
cffi @ file:///Users/sym/Library/Caches/pypoetry/artifacts/e6/3a/ed/211634ffd1f9ed0a6957a6981ef59f3af9934e62dc2ee15819d9623196/cffi-1.15.0-cp38-cp38-macosx_10_9_x86_64.whl
chardet @ file:///Users/sym/Library/Caches/pypoetry/artifacts/47/b7/82/19c2b887f87f3adbaf4e34c55189388e5132c78f6929d7001a78b0209b/chardet-4.0.0-py2.py3-none-any.whl
charset-normalizer @ file:///Users/sym/Library/Caches/pypoetry/artifacts/23/43/96/9c5bffd9765f723eb96925cec608e36125667e8f31b6b3152f0b51716f/charset_normalizer-2.0.7-py3-none-any.whl
click @ file:///Users/sym/Library/Caches/pypoetry/artifacts/3c/f6/a3/2262df8f5a6f3de5bbc78cc13803c60524c8384dce76661fdfe3df975f/click-8.0.3-py3-none-any.whl
colorlog @ file:///Users/sym/Library/Caches/pypoetry/artifacts/f5/a7/3e/1f1413ec92fc0a7c37b13f3d879599c82cd789f81c76e18089b6925b80/colorlog-6.6.0-py2.py3-none-any.whl
coverage @ file:///Users/sym/Library/Caches/pypoetry/artifacts/0e/17/32/a937f84a598e3a99f4e437f42df4b8e891e02c39aeefca49fdee9e4c86/coverage-6.1.1-cp38-cp38-macosx_10_9_x86_64.whl
cryptography @ file:///Users/sym/Library/Caches/pypoetry/artifacts/65/42/84/99e7508590fbc4796163f2f564ab4caf7c7a512127f55f9e0ef5b96d6d/cryptography-3.4.8-cp36-abi3-macosx_10_10_x86_64.whl
distlib @ file:///Users/sym/Library/Caches/pypoetry/artifacts/03/f1/f5/36347a1de773fbd629afb90a04cd682421c383092f800cd4b0ee90379a/distlib-0.3.3-py2.py3-none-any.whl
filelock @ file:///Users/sym/Library/Caches/pypoetry/artifacts/1c/30/62/c8f5fef72567efad2e65fd88a591a98c265774421bbd6a8e8ca4424c9b/filelock-3.3.2-py3-none-any.whl
frozenlist @ file:///Users/sym/Library/Caches/pypoetry/artifacts/c7/97/b2/2f3e0bc769592d27f0ff9dfa1d5c4d666a8e19a0ff758560892b13cab8/frozenlist-1.2.0-cp38-cp38-macosx_10_9_x86_64.whl
google-ads @ file:///Users/sym/Library/Caches/pypoetry/artifacts/be/74/8c/68441760850e611b65470fbf5cb0b23856284bbd3023e4ff9a231192f8/google_ads-14.1.0-py3-none-any.whl
google-api-core @ file:///Users/sym/Library/Caches/pypoetry/artifacts/6c/9f/cb/f29b967235fcf45553f3707e3b4075af2d12f393b250e9262e595cbc8f/google_api_core-2.2.2-py2.py3-none-any.whl
google-auth @ file:///Users/sym/Library/Caches/pypoetry/artifacts/7f/fe/8c/3b0b80802d984bbf73e991482e6e88c196d8deb2fd45306483de7b67e8/google_auth-1.35.0-py2.py3-none-any.whl
google-auth-oauthlib @ file:///Users/sym/Library/Caches/pypoetry/artifacts/35/f1/0a/8a2ac196dbf26e110b0c838bbbaeb3707a1475bae0e456bdf49b0dbb7b/google_auth_oauthlib-0.4.6-py2.py3-none-any.whl
googleapis-common-protos @ file:///Users/sym/Library/Caches/pypoetry/artifacts/8c/2e/ad/efdce99c62dbe56fda156dd62f807ea79a43f83dea45bd9f9ff4bc6240/googleapis_common_protos-1.53.0-py2.py3-none-any.whl
grpcio @ file:///Users/sym/Library/Caches/pypoetry/artifacts/7f/26/38/939f4ababe22e4d92e9206b0514c35e0803a063c66da76c941caca8c02/grpcio-1.41.1-cp38-cp38-macosx_10_10_x86_64.whl
idna @ file:///Users/sym/Library/Caches/pypoetry/artifacts/9e/f9/03/6066b92d35486e7f0d4f310126e3c60a6619726e43ef98e32e105b5c52/idna-2.10-py2.py3-none-any.whl
iniconfig @ file:///Users/sym/Library/Caches/pypoetry/artifacts/ac/f9/da/e990ffcd9ec361a68676a5916e391286e1ea5d1b8907ae887e141a71f5/iniconfig-1.1.1-py2.py3-none-any.whl
jmespath @ file:///Users/sym/Library/Caches/pypoetry/artifacts/43/d5/a2/f83573231324de7f5b61f5c607fbbe82ca535359a452de4852d2e25e8d/jmespath-0.10.0-py2.py3-none-any.whl
loguru @ file:///Users/sym/Library/Caches/pypoetry/artifacts/d0/16/90/b85b179e5e32051e734faa1aea1335c15d5c74447de12006487c70f02d/loguru-0.5.3-py3-none-any.whl
multidict @ file:///Users/sym/Library/Caches/pypoetry/artifacts/ce/07/97/96c8c0778bcefb790a0e0509f8f3114fc02f75a074477cac59c5174797/multidict-5.2.0-cp38-cp38-macosx_10_9_x86_64.whl
mypy-extensions @ file:///Users/sym/Library/Caches/pypoetry/artifacts/b6/a0/b0/a5dc9acd6fd12aba308634f21bb7cf0571448f20848797d7ecb327aa12/mypy_extensions-0.4.3-py2.py3-none-any.whl
nox @ file:///Users/sym/Library/Caches/pypoetry/artifacts/57/03/a0/2d9d2709981a7c7d46d9b81bc87f4487e266d5b5b7b6d26f1fdfbdc4ef/nox-2021.10.1-py3-none-any.whl
oauthlib @ file:///Users/sym/Library/Caches/pypoetry/artifacts/e6/64/ef/a43b939f04c9ee17174fb3657dba149ed15ba473ef44adf939f5032b85/oauthlib-3.1.1-py2.py3-none-any.whl
packaging @ file:///Users/sym/Library/Caches/pypoetry/artifacts/97/a9/40/465a52d3ec4e9927b21488ac9bf7c5fa59e9ee44c337de47d197b804e4/packaging-21.2-py3-none-any.whl
pathspec @ file:///Users/sym/Library/Caches/pypoetry/artifacts/04/91/14/294ed2ee6c852b0d466bdd15d393127eff4168b35ae81cedf5a03fe348/pathspec-0.9.0-py2.py3-none-any.whl
pep562 @ file:///Users/sym/Library/Caches/pypoetry/artifacts/e1/15/52/2820b854d72fc1a2f0816aae9ee4f803435ffcdae9a07aa30760afebdc/pep562-1.1-py2.py3-none-any.whl
platformdirs @ file:///Users/sym/Library/Caches/pypoetry/artifacts/c2/ba/64/b6513f98bf6524a0ec2316e6f9326829bf25522e915b9965b0e55b969b/platformdirs-2.4.0-py3-none-any.whl
pluggy @ file:///Users/sym/Library/Caches/pypoetry/artifacts/38/05/d7/70e2b0d553c780097b692ed658b0f553a28f16028d848a98174b1a2249/pluggy-1.0.0-py2.py3-none-any.whl
proto-plus @ file:///Users/sym/Library/Caches/pypoetry/artifacts/72/27/84/cb70b1de7be5e0fcb6fbf2f83f0b2dcf6b7c269401c84348de2446be52/proto_plus-1.18.1-py3-none-any.whl
protobuf @ file:///Users/sym/Library/Caches/pypoetry/artifacts/da/16/58/39025383d35aa8857945fb78e0716001e527e7dee801db8994234468da/protobuf-3.17.3-cp38-cp38-macosx_10_9_x86_64.whl
py @ file:///Users/sym/Library/Caches/pypoetry/artifacts/6b/b2/2b/e6686e7d0183dbd36bd66921efa3e77ce26260a3671524cd86614290e0/py-1.10.0-py2.py3-none-any.whl
pyasn1 @ file:///Users/sym/Library/Caches/pypoetry/artifacts/a6/f7/5e/59a43ec23ade0888631b3f24244da7f5d5a0b6b40849c86b8c6b4c54d1/pyasn1-0.4.8-py2.py3-none-any.whl
pyasn1-modules @ file:///Users/sym/Library/Caches/pypoetry/artifacts/09/f6/56/0e33158234b3e9b9e5d8eb01e7a96670a99ab43b3dbe89b0f129477a59/pyasn1_modules-0.2.8-py2.py3-none-any.whl
pycparser @ file:///Users/sym/Library/Caches/pypoetry/artifacts/f1/03/25/40eb46f7bede64f78ba073e2141b8216e611cbcde72e3117c326560101/pycparser-2.20-py2.py3-none-any.whl
pydantic @ file:///Users/sym/Library/Caches/pypoetry/artifacts/af/47/17/f8f37bb7abd91fed95dd4126981e6192c373440fdb36b85365c1917ac8/pydantic-1.7.4-cp38-cp38-macosx_10_9_x86_64.whl
PyJWT @ file:///Users/sym/Library/Caches/pypoetry/artifacts/7c/13/04/3e4aa78f6ee3ee721315e24346fe7226d1fc7330485447a1ce5d1151c7/PyJWT-2.3.0-py3-none-any.whl
PyMySQL @ file:///Users/sym/Library/Caches/pypoetry/artifacts/95/eb/3e/086ab3187a3284af82f8239d098217d060bca74ce9da2651bc5576a9c5/PyMySQL-0.10.1-py2.py3-none-any.whl
pyparsing @ file:///Users/sym/Library/Caches/pypoetry/artifacts/da/e7/3d/1780282f558e5fd157bf708b28b8ba0d08323ef6bc5b6396139ce38a0b/pyparsing-2.4.7-py2.py3-none-any.whl
pytest @ file:///Users/sym/Library/Caches/pypoetry/artifacts/90/e1/d5/985c1b371486cf5749ce1ec1beac04524dacb0642a52d916fd1aa82671/pytest-6.2.5-py3-none-any.whl
pytest-cov @ file:///Users/sym/Library/Caches/pypoetry/artifacts/b9/58/55/db0439b8e3e9c6ad7790a801ddb71803fd3791b5d6fd7f65efe27016f5/pytest_cov-2.10.1-py2.py3-none-any.whl
pytest-mock @ file:///Users/sym/Library/Caches/pypoetry/artifacts/30/d9/c2/bb39c4b3c8494e50e2dfa5a17991686a54be0256d6a30893fe796dc9fe/pytest_mock-3.4.0-py3-none-any.whl
python-dateutil @ file:///Users/sym/Library/Caches/pypoetry/artifacts/0f/f1/e2/e1d3399cea26388e2ed5b93ea3e7c137d2b4027c5ba14c64ab839294ed/python_dateutil-2.8.2-py2.py3-none-any.whl
pytz @ file:///Users/sym/Library/Caches/pypoetry/artifacts/07/c1/43/6f22a533e7bc5e277f10b4c6033e936ff7bbeb6464e5f27dbfb6c641d7/pytz-2021.3-py2.py3-none-any.whl
PyYAML @ file:///Users/sym/Library/Caches/pypoetry/artifacts/99/c6/60/2baceb8eb743927ffda4f0184e9aea5339ca72c9a463509feeeb0d1668/PyYAML-5.4.1-cp38-cp38-macosx_10_9_x86_64.whl
regex @ file:///Users/sym/Library/Caches/pypoetry/artifacts/d7/6f/34/ca74ab99f0547627cd097e864e92476bc96b3286da62684dd5b02dddbf/regex-2021.11.2-cp38-cp38-macosx_10_9_x86_64.whl
requests @ file:///Users/sym/Library/Caches/pypoetry/artifacts/fe/f0/74/d691c7619cd645682c495ff78949b9c284e88d3f2ee85124df5cec5b0a/requests-2.25.1-py2.py3-none-any.whl
tenacity @ file:///Users/sym/Library/Caches/pypoetry/artifacts/44/71/21/397279f8fca199d048595a0942e6b22859b07f13d4e7496ddf932be65d/tenacity-8.0.1-py3-none-any.whl
toml @ file:///Users/sym/Library/Caches/pypoetry/artifacts/9f/8c/c3/7b4f6778d60f3c9fa11f8fd0e48243bbad25a04975e0a01006b6350594/toml-0.10.2-py2.py3-none-any.whl.3.0-py2.py3-none-any.whl
typed-ast @ file:///Users/sym/Library/Caches/pypoetry/artifacts/2a/97/67/6a4e444fede37f31a09025fea88c54d4a40d739c4640e72d4c8d355661/typed_ast-1.4.3-cp38-cp38-macosx_10_9_x86_64.whl
typing-extensions @ file:///Users/sym/Library/Caches/pypoetry/artifacts/63/00/66/294e7d75d34398c27c7ed9b154822f99422b0f11548b11c569c188eb2b/typing_extensions-3.10.0.2-py3-none-any.whl
urllib3 @ file:///Users/sym/Library/Caches/pypoetry/artifacts/d8/12/e2/1b499db5e41e88fd0f31d0e056a4dc7b53824f63f027fffc1702969c57/urllib3-1.26.7-py2.py3-none-any.whly.whl
virtualenv @ file:///Users/sym/Library/Caches/pypoetry/artifacts/f0/72/e5/03355e512a823b9fe1f8905fca1d21e958e08ee2501c03106460ae69a1/virtualenv-20.10.0-py2.py3-none-any.whl
yarl @ file:///Users/sym/Library/Caches/pypoetry/artifacts/f1/71/2d/1439e8877c108b7d845e8e4199efa1fc8c8f30c3d25cdaa912722eba93/yarl-1.7.2-cp38-cp38-macosx_10_9_x86_64.whl_14_x86_64.whl
Request/Response Logs:
[WARNING] 2021-12-07T19:03:11.733Z e853bb65-dd7d-5b95-9b9c-e14c6f59508a Request made: ClientCustomerId: **MASKING**, Host: googleads.googleapis.com, Method: /google.ads.googleads.v9.services.GoogleAdsService/SearchStream, RequestId: wPV3zIKhwoN0VcBuXFLZxA, IsFault: True, FaultMessage: Internal error encountered.
Anything else we should know about your project / environment:
Reacted by Vinay Hegde
Activity
BenRKarl commentedon Dec 13, 2021
@holyachon How often are you receiving INTERNAL errors for this query? Is it intermittent or consistent?
It looks like in your application
google-api-coreis having a problem importing thegrpciodependency. If you look at this line it setsgrpctoNoneif it can't import it, then later here the error is thrown. Would it be possible for you to inspect this file at runtime, either with a breakpoint or print statement, to see what the state of thegrpclibrary is when it's executed?holyachon commentedon Dec 17, 2021
@BenRKarl Hello. Thank you for your response.
It seems to happen once or twice a day. I think adding a breakpoint is difficult because this error cannot be reproduced at my own will.
Then would it be helpful to log the state of
grpcwhen an error occurs like the code below?androkon commentedon Dec 17, 2021
I think, this problem from this line, module
grpc_statusdoes not exists, and even if importing of modulegrpcis success, in except block the variablegrpcbecomesNone.May be, code must be as this?
from grpc import Status as rpc_statusandrokon commentedon Dec 21, 2021
It seems like helps explicit package setup
pip install grpcio-statusholyachon commentedon Dec 22, 2021
I tried it and I think it works well. (because after install grpcio-status, I can reference
rpc_status)I'll have to take a closer look, but I think this dependency should be added to googleapis/python-api-core.
There seems to be a bit of a related issue. googleapis/python-api-core#301
Thank you for your help @androkon :)
holyachon commentedon Dec 22, 2021
The Issue says grpcio-status is optional dependency in python-api-core.
So I think it should be added to google-ads-python.
May I ask what you think? @BenRKarl
BenRKarl commentedon Jan 27, 2022
@holyachon thanks for working on this and sharing these details. I'm hesitant to make
grpcio-statusa dependency of this project because it should be resolved withinpython-api-coresince that's the library where it's used, and from where the error originates. I'm going to close this issue and the related PR and will ping the maintainers of api core to try to find a resolution in their library via the issue here: googleapis/python-api-core#301In the meantime I recommend the temporary workaround of manually installing
grpcio-status.