Importing aiplatform fails after protobuf 4.21

[dragos-cojocari]

This started occurring today, May 26th after protobuf 4.21 has been launched. Trying to explicitly set protobuf version to 3.21.0 in the requirements file of the project importing aiplatform does not help either. The other Google Python APIs ( storage etc) do not seem affected by this.

Environment details

• OS type and version: Pyton3.9.13-slim-bullseye
• Python version: 3.9.,13
• pip version: 22.04
• google-cloud-aiplatform version: 1.13.0

Steps to reproduce

Build any Python code that imports google.aiplatform

Code example

from google.cloud import aiplatform

Stack trace

/tmp/.tox/py39/lib/python3.9/site-packages/google/cloud/aiplatform/__init__.py:41: in <module>
unit_1      |     from google.cloud.aiplatform.matching_engine import (
unit_1      | /tmp/.tox/py39/lib/python3.9/site-packages/google/cloud/aiplatform/matching_engine/__init__.py:26: in <module>
unit_1      |     from google.cloud.aiplatform.matching_engine.matching_engine_index_endpoint import (
unit_1      | /tmp/.tox/py39/lib/python3.9/site-packages/google/cloud/aiplatform/matching_engine/matching_engine_index_endpoint.py:30: in <module>
unit_1      |     from google.cloud.aiplatform.matching_engine._protos import match_service_pb2
unit_1      | /tmp/.tox/py39/lib/python3.9/site-packages/google/cloud/aiplatform/matching_engine/_protos/match_service_pb2.py:54: in <module>
unit_1      |     _descriptor.FieldDescriptor(
unit_1      | /tmp/.tox/py39/lib/python3.9/site-packages/google/protobuf/descriptor.py:560: in __new__
unit_1      |     _message.Message._CheckCalledFromGeneratedFile()
unit_1      | E   TypeError: Descriptors cannot not be created directly.
unit_1      | E   If this call came from a _pb2.py file, your generated code is out of date and must be regenerated with protoc >= 3.19.0.
unit_1      | E   If you cannot immediately regenerate your protos, some other possible workarounds are:
unit_1      | E    1. Downgrade the protobuf package to 3.20.x or lower.
unit_1      | E    2. Set PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python (but this will use pure-Python parsing and will be much slower).```

[dragos-cojocari]

Here is a simple end to end way to reproduce this in a fresh Conda Env

% conda create -n test-pb python=3.9
conda activate test-pb
pip install google-ai-platform
python
from google.cloud import aiplatform 

With output

% conda create -n test-pb python=3.9
Collecting package metadata (current_repodata.json): done
Solving environment: done


==> WARNING: A newer version of conda exists. <==
  current version: 4.10.3
  latest version: 4.12.0

Please update conda by running

    $ conda update -n base -c defaults conda



## Package Plan ##

  environment location: /Users/dragoscojocari/opt/anaconda3/envs/test-pb

  added / updated specs:
    - python=3.9


The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    ca-certificates-2022.4.26  |       hecd8cb5_0         124 KB
    certifi-2022.5.18.1        |   py39hecd8cb5_0         148 KB
    openssl-1.1.1o             |       hca72f7f_0         2.2 MB
    sqlite-3.38.3              |       h707629a_0         1.2 MB
    tk-8.6.11                  |       h3fd3227_1         3.0 MB
    xz-5.2.5                   |       hca72f7f_1         244 KB
    zlib-1.2.12                |       h4dc903c_2          94 KB
    ------------------------------------------------------------
                                           Total:         7.0 MB

The following NEW packages will be INSTALLED:

  ca-certificates    pkgs/main/osx-64::ca-certificates-2022.4.26-hecd8cb5_0
  certifi            pkgs/main/osx-64::certifi-2022.5.18.1-py39hecd8cb5_0
  libcxx             pkgs/main/osx-64::libcxx-12.0.0-h2f01273_0
  libffi             pkgs/main/osx-64::libffi-3.3-hb1e8313_2
  ncurses            pkgs/main/osx-64::ncurses-6.3-hca72f7f_2
  openssl            pkgs/main/osx-64::openssl-1.1.1o-hca72f7f_0
  pip                pkgs/main/osx-64::pip-21.2.4-py39hecd8cb5_0
  python             pkgs/main/osx-64::python-3.9.12-hdfd78df_0
  readline           pkgs/main/osx-64::readline-8.1.2-hca72f7f_1
  setuptools         pkgs/main/osx-64::setuptools-61.2.0-py39hecd8cb5_0
  sqlite             pkgs/main/osx-64::sqlite-3.38.3-h707629a_0
  tk                 pkgs/main/osx-64::tk-8.6.11-h3fd3227_1
  tzdata             pkgs/main/noarch::tzdata-2022a-hda174b7_0
  wheel              pkgs/main/noarch::wheel-0.37.1-pyhd3eb1b0_0
  xz                 pkgs/main/osx-64::xz-5.2.5-hca72f7f_1
  zlib               pkgs/main/osx-64::zlib-1.2.12-h4dc903c_2


Proceed ([y]/n)? y


Downloading and Extracting Packages
tk-8.6.11            | 3.0 MB    | ###################################################################################################### | 100% 
xz-5.2.5             | 244 KB    | ###################################################################################################### | 100% 
openssl-1.1.1o       | 2.2 MB    | ###################################################################################################### | 100% 
ca-certificates-2022 | 124 KB    | ###################################################################################################### | 100% 
certifi-2022.5.18.1  | 148 KB    | ###################################################################################################### | 100% 
sqlite-3.38.3        | 1.2 MB    | ###################################################################################################### | 100% 
zlib-1.2.12          | 94 KB     | ###################################################################################################### | 100% 
Preparing transaction: done
Verifying transaction: done
Executing transaction: done
#
# To activate this environment, use
#
#     $ conda activate test-pb
#
# To deactivate an active environment, use
#
#     $ conda deactivate

%  conda activate test-pb
(test-pb)  % pip install google-cloud-aiplatform
Collecting google-cloud-aiplatform
  Downloading google_cloud_aiplatform-1.13.0-py2.py3-none-any.whl (1.8 MB)
     |████████████████████████████████| 1.8 MB 1.5 MB/s 
Collecting google-cloud-bigquery<3.0.0dev,>=1.15.0
  Using cached google_cloud_bigquery-2.34.3-py2.py3-none-any.whl (206 kB)
Collecting google-cloud-resource-manager<3.0.0dev,>=1.3.3
  Downloading google_cloud_resource_manager-1.5.0-py2.py3-none-any.whl (230 kB)
     |████████████████████████████████| 230 kB 851 kB/s 
Collecting google-cloud-storage<3.0.0dev,>=1.32.0
  Using cached google_cloud_storage-2.3.0-py2.py3-none-any.whl (107 kB)
Collecting google-api-core[grpc]!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.0,<3.0.0dev,>=1.31.5
  Downloading google_api_core-2.8.0-py3-none-any.whl (114 kB)
     |████████████████████████████████| 114 kB 586 kB/s 
Collecting proto-plus>=1.15.0
  Downloading proto_plus-1.20.4-py3-none-any.whl (46 kB)
     |████████████████████████████████| 46 kB 1.9 MB/s 
Collecting packaging>=14.3
  Using cached packaging-21.3-py3-none-any.whl (40 kB)
Collecting protobuf>=3.12.0
  Downloading protobuf-4.21.0-cp37-abi3-macosx_10_9_universal2.whl (483 kB)
     |████████████████████████████████| 483 kB 1.8 MB/s 
Collecting googleapis-common-protos<2.0dev,>=1.52.0
  Downloading googleapis_common_protos-1.56.1-py2.py3-none-any.whl (211 kB)
     |████████████████████████████████| 211 kB 1.2 MB/s 
Collecting requests<3.0.0dev,>=2.18.0
  Using cached requests-2.27.1-py2.py3-none-any.whl (63 kB)
Collecting google-auth<3.0dev,>=1.25.0
  Using cached google_auth-2.6.6-py2.py3-none-any.whl (156 kB)
Collecting grpcio-status<2.0dev,>=1.33.2
  Downloading grpcio_status-1.46.3-py3-none-any.whl (10.0 kB)
Collecting grpcio<2.0dev,>=1.33.2
  Downloading grpcio-1.46.3-cp39-cp39-macosx_10_10_x86_64.whl (4.4 MB)
     |████████████████████████████████| 4.4 MB 1.4 MB/s 
Collecting cachetools<6.0,>=2.0.0
  Using cached cachetools-5.1.0-py3-none-any.whl (9.2 kB)
Collecting rsa<5,>=3.1.4
  Using cached rsa-4.8-py3-none-any.whl (39 kB)
Collecting pyasn1-modules>=0.2.1
  Using cached pyasn1_modules-0.2.8-py2.py3-none-any.whl (155 kB)
Collecting six>=1.9.0
  Using cached six-1.16.0-py2.py3-none-any.whl (11 kB)
Collecting google-cloud-core<3.0.0dev,>=1.4.1
  Using cached google_cloud_core-2.3.0-py2.py3-none-any.whl (29 kB)
Collecting google-resumable-media<3.0dev,>=0.6.0
  Downloading google_resumable_media-2.3.3-py2.py3-none-any.whl (76 kB)
     |████████████████████████████████| 76 kB 2.1 MB/s 
Collecting python-dateutil<3.0dev,>=2.7.2
  Using cached python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB)
Collecting grpc-google-iam-v1<1.0.0dev,>=0.12.3
  Using cached grpc_google_iam_v1-0.12.4-py2.py3-none-any.whl (26 kB)
Collecting google-crc32c<2.0dev,>=1.0
  Using cached google_crc32c-1.3.0-cp39-cp39-macosx_10_9_x86_64.whl (30 kB)
Collecting pyparsing!=3.0.5,>=2.0.2
  Downloading pyparsing-3.0.9-py3-none-any.whl (98 kB)
     |████████████████████████████████| 98 kB 1.3 MB/s 
Collecting pyasn1<0.5.0,>=0.4.6
  Using cached pyasn1-0.4.8-py2.py3-none-any.whl (77 kB)
Collecting charset-normalizer~=2.0.0
  Using cached charset_normalizer-2.0.12-py3-none-any.whl (39 kB)
Requirement already satisfied: certifi>=2017.4.17 in /Users/dragoscojocari/opt/anaconda3/envs/test-pb/lib/python3.9/site-packages (from requests<3.0.0dev,>=2.18.0->google-api-core[grpc]!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.0,<3.0.0dev,>=1.31.5->google-cloud-aiplatform) (2022.5.18.1)
Collecting idna<4,>=2.5
  Using cached idna-3.3-py3-none-any.whl (61 kB)
Collecting urllib3<1.27,>=1.21.1
  Using cached urllib3-1.26.9-py2.py3-none-any.whl (138 kB)
Installing collected packages: pyasn1, urllib3, six, rsa, pyasn1-modules, protobuf, idna, charset-normalizer, cachetools, requests, grpcio, googleapis-common-protos, google-auth, pyparsing, grpcio-status, google-crc32c, google-api-core, python-dateutil, proto-plus, packaging, grpc-google-iam-v1, google-resumable-media, google-cloud-core, google-cloud-storage, google-cloud-resource-manager, google-cloud-bigquery, google-cloud-aiplatform
Successfully installed cachetools-5.1.0 charset-normalizer-2.0.12 google-api-core-2.8.0 google-auth-2.6.6 google-cloud-aiplatform-1.13.0 google-cloud-bigquery-2.34.3 google-cloud-core-2.3.0 google-cloud-resource-manager-1.5.0 google-cloud-storage-2.3.0 google-crc32c-1.3.0 google-resumable-media-2.3.3 googleapis-common-protos-1.56.1 grpc-google-iam-v1-0.12.4 grpcio-1.46.3 grpcio-status-1.46.3 idna-3.3 packaging-21.3 proto-plus-1.20.4 protobuf-4.21.0 pyasn1-0.4.8 pyasn1-modules-0.2.8 pyparsing-3.0.9 python-dateutil-2.8.2 requests-2.27.1 rsa-4.8 six-1.16.0 urllib3-1.26.9
(test-pb)  % python
Python 3.9.12 (main, Apr  5 2022, 01:53:17) 
[Clang 12.0.0 ] :: Anaconda, Inc. on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from google.cloud import aiplatform
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/dragoscojocari/opt/anaconda3/envs/test-pb/lib/python3.9/site-packages/google/cloud/aiplatform/__init__.py", line 41, in <module>
    from google.cloud.aiplatform.matching_engine import (
  File "/Users/dragoscojocari/opt/anaconda3/envs/test-pb/lib/python3.9/site-packages/google/cloud/aiplatform/matching_engine/__init__.py", line 26, in <module>
    from google.cloud.aiplatform.matching_engine.matching_engine_index_endpoint import (
  File "/Users/dragoscojocari/opt/anaconda3/envs/test-pb/lib/python3.9/site-packages/google/cloud/aiplatform/matching_engine/matching_engine_index_endpoint.py", line 30, in <module>
    from google.cloud.aiplatform.matching_engine._protos import match_service_pb2
  File "/Users/dragoscojocari/opt/anaconda3/envs/test-pb/lib/python3.9/site-packages/google/cloud/aiplatform/matching_engine/_protos/match_service_pb2.py", line 54, in <module>
    _descriptor.FieldDescriptor(
  File "/Users/dragoscojocari/opt/anaconda3/envs/test-pb/lib/python3.9/site-packages/google/protobuf/descriptor.py", line 560, in __new__
    _message.Message._CheckCalledFromGeneratedFile()
TypeError: Descriptors cannot not be created directly.
If this call came from a _pb2.py file, your generated code is out of date and must be regenerated with protoc >= 3.19.0.
If you cannot immediately regenerate your protos, some other possible workarounds are:
 1. Downgrade the protobuf package to 3.20.x or lower.
 2. Set PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python (but this will use pure-Python parsing and will be much slower).

More information: https://developers.google.com/protocol-buffers/docs/news/2022-05-06#python-updates
>>>