Can I start the another cluster from current notebook in Databricks?

Question

I have notebook1 assigned to cluster1 and notebook2 assigned to cluster2. I want to trigger notebook2 from notebook1 but notebook2 should use only cluster2 for execution. Currently its getting triggered using Cluster1. Please let me know for more information.

Answer 1

Unfortunately, you cannot start another cluster from current notebook.

This is excepted behaviour, when you trigger notebook2 from notebook it will use cluster1 and not cluster2.

Reason: When you run any command from notebook1, always runs on the attached cluster.

Notebooks cannot be statically assigned to a cluster; that's actually runtime state only. If you want to run some code on a different cluster (in this case, the code is a notebook), then you have to do it by having your first notebook submit a separate job, rather than using dbutils.notebook.run or %run.

Notebook Job Details:

Hope this helps.