How to migrate MlFlow experiments from one Databricks workspace to another with registered models?

Question

so unfortunatly we have to redeploy our Databricks Workspace in which we use the MlFlow functonality with the Experiments and the registering of Models.

However if you export the user folder where the eyperiment is saved with a DBC and import it into the new workspace, the Experiments are not migrated and are just missing.

So the easiest solution did not work. The next thing I tried was to create a new experiment in the new workspace. Copy all the experiment data from the dbfs of the old workspace (with dbfs cp -r dbfs:/databricks/mlflow source, and then the same again to upload it to the new workspace) to the new one. And then just reference the location of the data to the experiment like in the following picture:

This is also not working, no run is visible, although the path is already existing.

The next idea was that the registred models are the most important one so at least those should be there and accessible. For that I used the documentation here: https://www.mlflow.org/docs/latest/model-registry.html.

With the following code you get a list of the registred models on the old workspace with the reference on the run_id and location.

from mlflow.tracking import MlflowClient

client = MlflowClient()
for rm in client.list_registered_models():
    pprint(dict(rm), indent=4)

And with this code you can add models to a model registry with a reference to the location of the artifact data (on the new workspace):

# first the general model must be defined
client.create_registered_model(name='MyModel')

# and then the run of the model you want to registre will be added to the model as version one
client.create_model_version( name='MyModel', run_id='9fde022012046af935fe52435840cf1', source='dbfs:/databricks/mlflow/experiment_id/run_id/artifacts/model')

But that did also not worked out. if you go into the Model Registry you get a message like this: .

And I really checked, at the given path (the source) there the data is really uploaded and also a model is existing.

Do you have any new ideas to migrate those models in Databricks?

Answer 1

There is no official way to migrate experiments from one workspace to another. However, leveraging the MLflow API, there is an "unofficial" tool that can migrate experiments minus the notebook revision associated with a run.

mlflow-tools