Parallelize HTTP requests with Pyspark

Question

I'm using pyspark to do some data transformation: like the following :

df_systems_tree_users = sqlContext.read.format("jdbc") \
    .option("dbtable",
            "(select ID as SYSTEMUID,M_EXTERNAL_ID,metric,DATATRANSMISSIONFREQUENCY,MODEL,BRAND,BUILDING FROM SYSTEM  INNER JOIN SENSOR ON SYSTEM.ID=SENSOR.SYSTEMID WHERE LPWANOPERATOR='Objenious' AND M_EXTERNAL_ID!='None' )") \
    .option("url", "jdbc:phoenix:master1:2181:/hbase-unsecure") \
    .option("driver", "org.apache.phoenix.jdbc.PhoenixDriver") \
    .load()
objRDD = df_systems_tree_users.rdd.map(lambda x: getStatesAndUplink(x))

getStatesAndUplink method uses requests python library to execute an http get request to an external api.

I run this spark job on 4 executors with 4 cores each , but it takes alot of time to run 30 minutes .

My question is how to optimize my code to parallelize in a efficient way my http requests ?

Answer 1

As mentioned in the documentation, you must specify 4 arguments:

• partitionColumn,
• lowerBound,
• upperBound,
• numPartitions

Only with those options Spark will parallelize reading - in other cases, it will be done in one thread.

Edit: There exist Phoenix Spark plugin. It will parallelize reading without specifying those 4 arguments.

Edit 2: Method getStatesAndUplink may be throttling external service if the data is unbalanced. It may just "stuck" on processing on one node