Repository navigation
ModelBuilder silently drops source_code when model data is a PipelineVariable #6450
Description
PySDK Version
- PySDK V2 (2.x)
- PySDK V3 (3.x)
Describe the bug
ModelBuilder.build() silently drops source_code when s3_model_data_url is a pipeline variable.
This combination is required when a pipeline creates a model from the output of a training step:
s3_model_data_url=<PipelineVariable>source_code=SourceCode(...)sagemaker_session=<PipelineSession>
The generatedCreateModelrequest contains the pipeline-variable model artifact, but its container environment is empty.
ModelStepalso contains no runtime repack step.
As a result, script-mode inference containers cannot find the entry script.
The same configuration works whens3_model_data_urlis a concrete S3 URI.
To reproduce
- Configure AWS credentials and a SageMaker execution role.
- Set the role ARN:
export SAGEMAKER_ROLE_ARN="arn:aws:iam::<account-id>:role/<sagemaker-role>"
export AWS_REGION="us-east-1"- Save the following script as
reproduce.py. - Run
python reproduce.py.
import os
import tempfile
from pathlib import Path
import boto3
from sagemaker.core import image_uris
from sagemaker.core.training.configs import SourceCode
from sagemaker.core.workflow.parameters import ParameterString
from sagemaker.core.workflow.pipeline_context import PipelineSession
from sagemaker.mlops.workflow.model_step import ModelStep
from sagemaker.serve import ModelBuilder
region = os.environ.get("AWS_REGION", "us-east-1")
role_arn = os.environ["SAGEMAKER_ROLE_ARN"]
boto_session = boto3.Session(region_name=region)
pipeline_session = PipelineSession(boto_session=boto_session)
image_uri = image_uris.retrieve(
framework="sklearn",
region=region,
version="1.4-2-py312",
py_version="py3",
instance_type="ml.m5.large",
image_scope="inference",
)
model_data = ParameterString(
name="ModelDataUrl",
default_value="s3://example-bucket/model.tar.gz",
)
with tempfile.TemporaryDirectory() as source_dir:
entry_script = Path(source_dir) / "inference.py"
entry_script.write_text(
"""
def model_fn(model_dir):
return None
def input_fn(request_body, content_type):
return request_body
def predict_fn(data, model):
return data
def output_fn(prediction, accept):
return prediction
""".strip()
)
builder = ModelBuilder(
image_uri=image_uri,
s3_model_data_url=model_data,
source_code=SourceCode(
source_dir=source_dir,
entry_script="inference.py",
),
role_arn=role_arn,
instance_type="ml.m5.large",
sagemaker_session=pipeline_session,
)
model_step = ModelStep(
name="CreateModel",
step_args=builder.build(model_name="pipeline-source-code-repro"),
)
primary_container = model_step.arguments["PrimaryContainer"]
environment = primary_container.get("Environment", {})
print("PrimaryContainer environment:", environment)
print("Runtime repack step count:", len(model_step.steps))
assert environment == {}, environment
assert len(model_step.steps) == 0
print("Bug reproduced: source_code was silently dropped.")Actual output:
PrimaryContainer environment: {}
Runtime repack step count: 0
Bug reproduced: source_code was silently dropped.
No SageMaker model or pipeline execution is created by this reproduction.
Expected behavior
ModelBuilder must preserve source_code when s3_model_data_url is a pipeline variable.
The generated ModelStep must add a runtime repack step that:
- Reads the model artifact after the training step completes.
- Adds
inference.pyundercode/. - Creates a self-contained
model.tar.gz. - Sets
SAGEMAKER_PROGRAM. - Sets
SAGEMAKER_SUBMIT_DIRECTORYto/opt/ml/model/code.
If pipeline-variable repacking is not supported, ModelBuilder.build() must raise a clear error. It must not silently remove the source code.
Screenshots or logs
The generated primary container has no script-mode environment:
PrimaryContainer environment: {}
Runtime repack step count: 0
The resulting CreateModel request contains the model-data pipeline expression but no inference-code configuration.
System information
- SageMaker Python SDK version:
sagemaker==3.24.0 - Component versions:
sagemaker-core==2.24.0sagemaker-serve==1.24.0sagemaker-mlops==1.24.0
- Framework name (eg. PyTorch) or algorithm (eg. KMeans): Scikit-learn
- Framework version:
1.4-2-py312 - Python version: 3.12
- CPU or GPU: CPU
- Custom Docker image (Y/N): N
- Operating system: macOS, Darwin 25.6.0
Additional context
The concrete-S3 version of this problem was fixed by: fix(serve): repack source_code for image_uri/ModelTrainer builds #6112
The v3.24 passthrough implementation recognizes a model artifact only when it is a string that starts withs3://. APipelineVariabledoes not satisfy that check.
The passthrough branch then clears:
self.source_dir = None
self.entry_point = None
self.dependencies = None
self.git_config = NoneThe SDK already contains pipeline-variable runtime-repack logic in _upload_code(), and ModelStep supports _RepackModelStep.
That logic is not reached from this passthrough path.
The existing integration test covers a concrete S3 model URI, but it does not cover a model-data pipeline variable: https://github.com/aws/sagemaker-python-sdk/blob/v3.24.0/sagemaker-serve/tests/integ/test_passthrough_source_code_repack_integration.py
Current workaround:
- Upload the inference source as a separate S3 archive.
- Pass
SAGEMAKER_PROGRAMandSAGEMAKER_SUBMIT_DIRECTORYthroughenv_vars.
This workaround is acceptable for a transient batch-transform model. It does not produce the self-contained artifact required for
later model-package registration.
Activity