Azure Databricks Notebook Parameter Passing

Question

I invoked a Child Notebook from a Parent Notebook with some parameters passed through.

I encountered limitations in the number of characters while passing parameters. As part of parameter passing, Table Schema (Name CHAR(5), Age INT etc.,) is passed to the child notebook.

The problem is that my table has 450 columns and the child notebook does not support passing parameters with more characters.

The error:

com.databricks.WorkflowException: com.databricks.common.client.DatabricksServiceException: INVALID_PARAMETER_VALUE: The provided parameters (in json representation) are too long (10046 bytes); it cannot exceeds 10000 bytes.

Answer 1

This is a known limitation with Azure Databricks.

While passing the parameters - The JSON representation of this field cannot exceed 10,000 bytes.

In case, if it exceeds the limit it will throws the above error message which you are experiencing.