feat: Vertex AI Experiments GA by sasha-gitg · Pull Request #1410 · googleapis/python-aiplatform

sasha-gitg · 2022-06-06T16:50:58Z

No description provided.

snippet-bot · 2022-06-06T16:51:13Z

Here is the summary of changes.

You are about to add 20 region tags.

samples/model-builder/experiment_tracking/assign_artifact_as_execution_input_sample.py:18, tag aiplatform_sdk_assign_artifact_as_execution_input_sample
samples/model-builder/experiment_tracking/assign_artifact_as_execution_output_sample.py:18, tag aiplatform_sdk_assign_artifact_as_execution_output_sample
samples/model-builder/experiment_tracking/create_artifact_sample.py:20, tag aiplatform_sdk_create_artifact_sample
samples/model-builder/experiment_tracking/create_experiment_run_sample.py:20, tag aiplatform_sdk_create_experiment_run_sample
samples/model-builder/experiment_tracking/create_experiment_sample.py:20, tag aiplatform_sdk_create_experiment_sample
samples/model-builder/experiment_tracking/delete_experiment_run_sample.py:20, tag aiplatform_sdk_delete_experiment_run_sample
samples/model-builder/experiment_tracking/delete_experiment_sample.py:19, tag aiplatform_sdk_delete_experiment_sample
samples/model-builder/experiment_tracking/end_experiment_run_sample.py:19, tag aiplatform_sdk_end_experiment_run_sample
samples/model-builder/experiment_tracking/get_artifact_sample.py:18, tag aiplatform_sdk_get_artifact_sample
samples/model-builder/experiment_tracking/get_experiment_data_frame_sample.py:19, tag aiplatform_sdk_get_experiments_data_frame_sample
samples/model-builder/experiment_tracking/get_experiment_run_metrics_sample.py:20, tag aiplatform_sdk_get_experiment_run_metrics_sample
samples/model-builder/experiment_tracking/get_experiment_run_params_sample.py:20, tag aiplatform_sdk_get_experiment_run_params_sample
samples/model-builder/experiment_tracking/get_experiment_run_time_series_metric_data_frame_sample.py:20, tag aiplatform_sdk_get_experiment_run_time_series_metric_data_frame_sample
samples/model-builder/experiment_tracking/log_metrics_sample.py:20, tag aiplatform_sdk_log_metrics_sample
samples/model-builder/experiment_tracking/log_params_sample.py:20, tag aiplatform_sdk_log_params_sample
samples/model-builder/experiment_tracking/log_pipeline_job_sample.py:19, tag aiplatform_sdk_log_pipeline_job_sample
samples/model-builder/experiment_tracking/log_pipeline_job_to_experiment_sample.py:20, tag aiplatform_sdk_log_pipeline_job_to_experiment_sample
samples/model-builder/experiment_tracking/log_time_series_metrics_sample.py:21, tag aiplatform_sdk_log_time_series_metrics_sample
samples/model-builder/experiment_tracking/resume_experiment_run_sample.py:19, tag aiplatform_sdk_resume_experiment_run_sample
samples/model-builder/experiment_tracking/start_execution_sample.py:20, tag aiplatform_sdk_start_execution_sample

This comment is generated by snippet-bot.
If you find problems with this result, please file an issue at:
https://github.com/googleapis/repo-automation-bots/issues.
To update this comment, add snippet-bot:force-run label or use the checkbox below:

Refresh this comment

…platform into experiments-release

sararob

Here's my review of the Artifact class.

samgoodman

Small fixes and suggestions, overall LGTM

sararob

Review of Context, Execution, Experiment Resources, and some tests.

jialuzh · 2022-06-07T23:08:37Z

+
+class _VertexResourceArtifactResolver:
+
+    _resource_to_artifact_type = {models.Model: "google.VertexModel"}


IIUC, this resolver would automatically link a resource to a MLMD artifact? And if so, is Model the only one we would support for GA? How about managed dataset?

I opened a ticket and added TODO for managed dataset. Let me sync with the service team to see if we can get this in before GA.

sararob · 2022-06-09T15:53:54Z

+
+    @staticmethod
+    def _get_experiment(
+        experiment: Optional[Union[experiment_resources.Experiment, str]] = None,


Flagging since in the docstring this is called experiment_name and is required.

sararob

My last batch of review, great job!

sararob · 2022-06-09T20:04:00Z


 from google.cloud.aiplatform import initializer
-from google.cloud.aiplatform.metadata.metadata import metadata_service
+from google.cloud.aiplatform.metadata.metadata import _experiment_tracker


Here's the error when I try to run test_initializer:

__________________________________________________________________________________________________________ ERROR collecting tests/unit/aiplatform/test_initializer.py __________________________________________________________________________________________________________ ImportError while importing test module '/Users/sararob/Dev/sara-fork/python-aiplatform/tests/unit/aiplatform/test_initializer.py'. Hint: make sure your test modules/packages have valid Python names. Traceback: /Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/importlib/__init__.py:127: in import_module return _bootstrap._gcd_import(name[level:], package, level) tests/unit/aiplatform/test_initializer.py:28: in <module> from google.cloud.aiplatform.metadata.metadata import _experiment_tracker E ImportError: cannot import name '_experiment_tracker' from 'google.cloud.aiplatform.metadata.metadata' (/Users/sararob/Library/Python/3.8/lib/python/site-packages/google/cloud/aiplatform/metadata/metadata.py)

Ark-kun · 2023-01-11T04:24:07Z

+                for key, value in metadata.items():
+                    # Note: This only support nested dictionaries one level deep
+                    if isinstance(value, collections.Mapping):
+                        gca_resource.metadata[key].update(value)


What is the proposed way to remove keys from a nested dictionary?

P.S. This also seems to fail when the key does not already exist in the metadata.

…ing metadata store init Fixes googleapis#6610. Two bugs in `metadata/execution.py` caused `Execution.create()` to fail with "503 Getting metadata from plugin failed with error: before_request": 1. The `credentials` parameter on both `create()` and `_create()` used `=` instead of `:` for the type annotation: `credentials=Optional[auth_credentials.Credentials]` instead of `credentials: Optional[auth_credentials.Credentials] = None`. This made the default value the `typing.Optional` type object itself (not `None`), which the gRPC auth stack tried to call `.before_request()` on. 2. `create()` skipped the `ensure_default_metadata_store_exists()` call that `Artifact.create()` makes, so standalone `Execution.create()` would also fail if no prior operation had initialized the metadata store. Both issues have been present since PR googleapis#1410 (June 2022). `artifact.py` and `context.py` both have the correct annotation.

sasha-gitg requested a review from a team June 6, 2022 16:50

sasha-gitg requested a review from a team as a code owner June 6, 2022 16:50

sasha-gitg requested a review from dandhlee June 6, 2022 16:50

product-auto-label Bot added size: xl Pull request size is extra large. api: vertex-ai Issues related to the googleapis/python-aiplatform API. labels Jun 6, 2022

sasha-gitg force-pushed the experiments-release branch 10 times, most recently from c0f099d to 4609ebb Compare June 7, 2022 17:26

sasha-gitg added 2 commits June 7, 2022 13:26

feat: Vertex AI Experiments GA

4609ebb

Merge branch 'main' into experiments-release

5bbb020

sasha-gitg requested review from jialuzh, samgoodman and sararob June 7, 2022 20:39

sasha-gitg added 2 commits June 7, 2022 16:54

chore: remove unused metadata schema implementation

c91a314

Merge branch 'experiments-release' of github.com:sasha-gitg/python-ai…

2a46205

…platform into experiments-release

sasha-gitg force-pushed the experiments-release branch 2 times, most recently from 040c59f to 786bd01 Compare June 8, 2022 13:41

chore: remove unused tensorboard run dev schema utils

978ecb4

sasha-gitg force-pushed the experiments-release branch from 786bd01 to 978ecb4 Compare June 8, 2022 13:52

jialuzh requested a review from SinaChavoshi June 8, 2022 17:43

sararob requested changes Jun 8, 2022

View reviewed changes

chore: address reviewer comments on artifact.py

96032ec

Merge branch 'main' into experiments-release

11c9fe0

samgoodman suggested changes Jun 8, 2022

View reviewed changes

sasha-gitg added the do not merge Indicates a pull request not ready for merge, due to either quality or timing. label Jun 9, 2022

chore: address review comments

37aa6ad

sararob requested changes Jun 9, 2022

View reviewed changes

sasha-gitg removed the do not merge Indicates a pull request not ready for merge, due to either quality or timing. label Jun 9, 2022

jialuzh reviewed Jun 9, 2022

View reviewed changes

sararob requested changes Jun 9, 2022

View reviewed changes

sasha-gitg added 2 commits June 9, 2022 13:39

chore: sample tests

91608dc

chore: resolve review comments

e549cba

sararob requested changes Jun 10, 2022

View reviewed changes

sasha-gitg and others added 5 commits June 10, 2022 12:41

chore: address reviewer comments

3c47349

chore: address reviewer comments

d55c9ff

chore: address reviewer comments

9e1b191

chore: address reviewer comments

9d0d5b3

Merge branch 'main' into experiments-release

905aeca

samgoodman approved these changes Jun 10, 2022

View reviewed changes

sasha-gitg added 2 commits June 13, 2022 10:10

chore: update samples tests and fix artifact unit tests

2432789

chore: add sample tests and lint

58e426f

sasha-gitg force-pushed the experiments-release branch from 8c5cde5 to 58e426f Compare June 13, 2022 17:42

sararob approved these changes Jun 13, 2022

View reviewed changes

dizcology approved these changes Jun 13, 2022

View reviewed changes

jialuzh approved these changes Jun 13, 2022

View reviewed changes

sasha-gitg merged commit 24d1bb6 into googleapis:main Jun 13, 2022

release-please Bot mentioned this pull request Jun 13, 2022

chore(main): release 1.15.0 #1419

Merged

Ark-kun reviewed Jan 11, 2023

View reviewed changes

release-please Bot mentioned this pull request Jun 8, 2023

chore(main): release 1.24.1 #2196

Closed

javabrett mentioned this pull request Apr 19, 2026

Execution.create() fails with 503 due to credentials annotation typo #6610

Open

javabrett mentioned this pull request Apr 19, 2026

fix: Execution.create() 503 from credentials annotation typo and missing metadata store init #6611

Open

3 tasks


		class _VertexResourceArtifactResolver:

		_resource_to_artifact_type = {models.Model: "google.VertexModel"}

Conversation

sasha-gitg commented Jun 6, 2022

Uh oh!

snippet-bot Bot commented Jun 6, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sararob left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

samgoodman left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

sararob left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jialuzh Jun 7, 2022

Choose a reason for hiding this comment

Uh oh!

sasha-gitg Jun 10, 2022

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

sararob Jun 9, 2022

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

sararob left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

sararob Jun 9, 2022

Choose a reason for hiding this comment

snippet-bot Bot commented Jun 6, 2022 •

edited

Loading