Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Execute python scripts in Azure DataFactory

I have my data stored in blobs and I have written a python script to do some computations and create another csv. How can I execute this in Azure Data Factory ?

like image 496
Mighty Avatar asked Dec 10 '22 05:12

Mighty


2 Answers

Mighty. You could use Azure Data Factory V2 custom activity for your requirements. You can directly execute a command to invoke python script using Custom Activity.

Please refer to this sample on the github.

Hope it helps you.

like image 102
Jay Gong Avatar answered Dec 12 '22 22:12

Jay Gong


Another option is using a DatabricksSparkPython Activity. This makes sense if you want to scale out, but could require some code modifications for PySpark support. Prerequisite of cause is an Azure Databricks workspace. You have to upload your script to DBFS and can trigger it via Azure Data Factory. The following example triggers the script pi.py:

{
    "activity": {
        "name": "MyActivity",
        "description": "MyActivity description",
        "type": "DatabricksSparkPython",
        "linkedServiceName": {
            "referenceName": "MyDatabricksLinkedservice",
             "type": "LinkedServiceReference"
        },
        "typeProperties": {
            "pythonFile": "dbfs:/docs/pi.py",
            "parameters": [
                "10"
            ],
            "libraries": [
                {
                    "pypi": {
                        "package": "tensorflow"
                    }
                }
            ]
        }
    }
}

See the Documentation for more details.

like image 23
Hauke Mallow Avatar answered Dec 12 '22 23:12

Hauke Mallow