Can spark-submit with named argument?

Question

I know i can pass argument to main function by

spark-submit com.xxx.test 1 2

and get argument by:

def main(args: Array[String]): Unit = {
    // 读取参数
    var city = args(0)
    var num = args(1)

but i want to know is there a path to pass named argument like:

spark-submit com.xxx.test --citys=1 --num=2

and how to get this named argument in main.scala?

Answer 1

you can write your own custom class which parses the input arguments based on the key something like below:

object CommandLineUtil {

  def getOpts(args: Array[String], usage: String): collection.mutable.Map[String, String] = {
    if (args.length == 0) {
      log.warn(usage)
      System.exit(1)
    }

    val (opts, vals) = args.partition {
      _.startsWith("-")
    }

    val optsMap = collection.mutable.Map[String, String]()
    opts.map { x =>
      val pair = x.split("=")
      if (pair.length == 2) {
        optsMap += (pair(0).split("-{1,2}")(1) -> pair(1))
      } else {
        log.warn(usage)
        System.exit(1)
      }
    }

    optsMap
  }
}

Then you can use the methods with in your spark application

val usage = "Usage:  [--citys] [--num]"
val optsMap = CommandLineUtil.getOpts(args, usage)
val citysValue = optsMap("citys")
val numValue = optsMap("num")

You can improvise CommandLineUtil as per your requirements