Skip to content

ModelBuilder silently drops source_code when model data is a PipelineVariable #6450

Open

Description

@Alex-Mesmer

PySDK Version

  • PySDK V2 (2.x)
  • PySDK V3 (3.x)

Describe the bug
ModelBuilder.build() silently drops source_code when s3_model_data_url is a pipeline variable.
This combination is required when a pipeline creates a model from the output of a training step:

  • s3_model_data_url=<PipelineVariable>
  • source_code=SourceCode(...)
  • sagemaker_session=<PipelineSession>
    The generated CreateModel request contains the pipeline-variable model artifact, but its container environment is empty.
    ModelStep also contains no runtime repack step.
    As a result, script-mode inference containers cannot find the entry script.
    The same configuration works when s3_model_data_url is a concrete S3 URI.

To reproduce

  1. Configure AWS credentials and a SageMaker execution role.
  2. Set the role ARN:
export SAGEMAKER_ROLE_ARN="arn:aws:iam::<account-id>:role/<sagemaker-role>"
export AWS_REGION="us-east-1"
  1. Save the following script as reproduce.py.
  2. Run python reproduce.py.
import os
import tempfile
from pathlib import Path

import boto3

from sagemaker.core import image_uris
from sagemaker.core.training.configs import SourceCode
from sagemaker.core.workflow.parameters import ParameterString
from sagemaker.core.workflow.pipeline_context import PipelineSession
from sagemaker.mlops.workflow.model_step import ModelStep
from sagemaker.serve import ModelBuilder

region = os.environ.get("AWS_REGION", "us-east-1")
role_arn = os.environ["SAGEMAKER_ROLE_ARN"]

boto_session = boto3.Session(region_name=region)
pipeline_session = PipelineSession(boto_session=boto_session)

image_uri = image_uris.retrieve(
    framework="sklearn",
    region=region,
    version="1.4-2-py312",
    py_version="py3",
    instance_type="ml.m5.large",
    image_scope="inference",
)

model_data = ParameterString(
    name="ModelDataUrl",
    default_value="s3://example-bucket/model.tar.gz",
)

with tempfile.TemporaryDirectory() as source_dir:
    entry_script = Path(source_dir) / "inference.py"
    entry_script.write_text(
        """
def model_fn(model_dir):
    return None

def input_fn(request_body, content_type):
    return request_body

def predict_fn(data, model):
    return data

def output_fn(prediction, accept):
    return prediction
""".strip()
    )

    builder = ModelBuilder(
        image_uri=image_uri,
        s3_model_data_url=model_data,
        source_code=SourceCode(
            source_dir=source_dir,
            entry_script="inference.py",
        ),
        role_arn=role_arn,
        instance_type="ml.m5.large",
        sagemaker_session=pipeline_session,
    )

    model_step = ModelStep(
        name="CreateModel",
        step_args=builder.build(model_name="pipeline-source-code-repro"),
    )

    primary_container = model_step.arguments["PrimaryContainer"]
    environment = primary_container.get("Environment", {})

    print("PrimaryContainer environment:", environment)
    print("Runtime repack step count:", len(model_step.steps))

    assert environment == {}, environment
    assert len(model_step.steps) == 0

    print("Bug reproduced: source_code was silently dropped.")

Actual output:

PrimaryContainer environment: {}
Runtime repack step count: 0
Bug reproduced: source_code was silently dropped.

No SageMaker model or pipeline execution is created by this reproduction.

Expected behavior
ModelBuilder must preserve source_code when s3_model_data_url is a pipeline variable.
The generated ModelStep must add a runtime repack step that:

  1. Reads the model artifact after the training step completes.
  2. Adds inference.py under code/.
  3. Creates a self-contained model.tar.gz.
  4. Sets SAGEMAKER_PROGRAM.
  5. Sets SAGEMAKER_SUBMIT_DIRECTORY to /opt/ml/model/code.

If pipeline-variable repacking is not supported, ModelBuilder.build() must raise a clear error. It must not silently remove the source code.

Screenshots or logs
The generated primary container has no script-mode environment:

PrimaryContainer environment: {}
Runtime repack step count: 0

The resulting CreateModel request contains the model-data pipeline expression but no inference-code configuration.
System information

  • SageMaker Python SDK version: sagemaker==3.24.0
  • Component versions:
    • sagemaker-core==2.24.0
    • sagemaker-serve==1.24.0
    • sagemaker-mlops==1.24.0
  • Framework name (eg. PyTorch) or algorithm (eg. KMeans): Scikit-learn
  • Framework version: 1.4-2-py312
  • Python version: 3.12
  • CPU or GPU: CPU
  • Custom Docker image (Y/N): N
  • Operating system: macOS, Darwin 25.6.0
    Additional context
    The concrete-S3 version of this problem was fixed by: fix(serve): repack source_code for image_uri/ModelTrainer builds #6112
    The v3.24 passthrough implementation recognizes a model artifact only when it is a string that starts with s3://. A PipelineVariable does not satisfy that check.
    The passthrough branch then clears:
self.source_dir = None
self.entry_point = None
self.dependencies = None
self.git_config = None

The SDK already contains pipeline-variable runtime-repack logic in _upload_code(), and ModelStep supports _RepackModelStep.
That logic is not reached from this passthrough path.
The existing integration test covers a concrete S3 model URI, but it does not cover a model-data pipeline variable: https://github.com/aws/sagemaker-python-sdk/blob/v3.24.0/sagemaker-serve/tests/integ/test_passthrough_source_code_repack_integration.py

Current workaround:

  1. Upload the inference source as a separate S3 archive.
  2. Pass SAGEMAKER_PROGRAM and SAGEMAKER_SUBMIT_DIRECTORY through env_vars.
    This workaround is acceptable for a transient batch-transform model. It does not produce the self-contained artifact required for
    later model-package registration.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions