Sagemaker pipelines - popping out X from the pipeline since it will be overriden

Question

Since upgrading sagemaker to 2.173+ logs got a bit noisy. When evaluating the Sagemaker Pipeline, I'm seeing plenty of

WARNING:sagemaker.workflow.utilities:Popping out 'ProcessingJobName' from the pipeline definition by default since it will be overridden at pipeline execution time. Please utilize the PipelineDefinitionConfig to persist this field in the pipeline definition if desired.
WARNING:sagemaker.workflow.utilities:Popping out 'ProcessingJobName' from the pipeline definition by default since it will be overridden at pipeline execution time. Please utilize the PipelineDefinitionConfig to persist this field in the pipeline definition if desired.
WARNING:sagemaker.workflow.utilities:Popping out 'TrainingJobName' from the pipeline definition by default since it will be overridden at pipeline execution time. Please utilize the PipelineDefinitionConfig to persist this field in the pipeline definition if desired.
WARNING:sagemaker.workflow._utils:Popping out 'CertifyForMarketplace' from the pipeline definition since it will be overridden in pipeline execution time.
WARNING:sagemaker.workflow.utilities:Popping out 'ModelPackageName' from the pipeline definition by default since it will be overridden at pipeline execution time. Please utilize the PipelineDefinitionConfig to persist this field in the pipeline definition if desired.

with the emphasis on Please utilize the PipelineDefinitionConfig to persist this field in the pipeline definition if desired. However, I can't find how to actually use that PipelineDefinitionConfig in the documentation, nor any example on github.

Any references how to resolve these warning without just suppressing them?

Answer 1

The access management documentation for pipelines has a more thorough explanation:

Job prefixing is off by default. To opt into this feature, use the use_custom_job_prefix option of PipelineDefinitionConfig as shown in the following snippet:

# Create a definition configuration and toggle on custom prefixing
definition_config = PipelineDefinitionConfig(use_custom_job_prefix=True);

# Create a pipeline with a custom prefix
 pipeline = Pipeline(
     name="MyJobPrefixedPipeline",
     parameters=[...]
     steps=[...]
     pipeline_definition_config=definition_config
)

The warnings seem to be purely informational. You can opt into job prefixing using the example above to make them go away, or just ignore them.