Logo Questions Linux Laravel Mysql Ubuntu Git Menu
HTML CSS JAVASCRIPT SQL PYTHON PHP BOOTSTRAP JAVA JQUERY R React Kotlin
 

Hive tables not found when running in YARN-Cluster mode

Tags:

apache-spark

apache-spark-sql

hive

hadoop-yarn

hortonworks-data-platform

I have a Spark (version 1.4.1) application on HDP 2.3. It works fine when running it in YARN-Client mode. However, when running it on YARN-Cluster mode none of my Hive tables can be found by the application.

I submit the application like so:

./bin/spark-submit 
    --class com.myCompany.Main 
    --master yarn-cluster 
    --num-executors 3 
    --driver-memory 4g 
    --executor-memory 10g 
    --executor-cores 1 
    --jars lib/datanucleus-api-jdo-3.2.6.jar,lib/datanucleus-rdbms-3.2.9.jar,lib/datanucleus-core-3.2.10.jar /home/spark/apps/YarnClusterTest.jar  
    --files /etc/hive/conf/hive-site.xml

Here's a excerpt from the logs:

5/12/02 11:05:13 INFO hive.HiveContext: Initializing execution hive, version 0.13.1
15/12/02 11:05:14 INFO metastore.HiveMetaStore: 0: Opening raw store with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore
15/12/02 11:05:14 INFO metastore.ObjectStore: ObjectStore, initialize called
15/12/02 11:05:14 INFO DataNucleus.Persistence: Property hive.metastore.integral.jdo.pushdown unknown - will be ignored
15/12/02 11:05:14 INFO DataNucleus.Persistence: Property datanucleus.cache.level2 unknown - will be ignored
15/12/02 11:05:14 INFO storage.BlockManagerMasterEndpoint: Registering block manager worker2.xxx.com:34697 with 5.2 GB RAM, BlockManagerId(1, worker2.xxx.com, 34697)
15/12/02 11:05:16 INFO metastore.ObjectStore: Setting MetaStore object pin classes with hive.metastore.cache.pinobjtypes="Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order"
15/12/02 11:05:16 INFO metastore.MetaStoreDirectSql: MySQL check failed, assuming we are not on mysql: Lexical error at line 1, column 5.  Encountered: "@" (64), after : "".
15/12/02 11:05:17 INFO DataNucleus.Datastore: The class "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as "embedded-only" so does not have its own datastore table.
15/12/02 11:05:17 INFO DataNucleus.Datastore: The class "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as "embedded-only" so does not have its own datastore table.
15/12/02 11:05:18 INFO DataNucleus.Datastore: The class "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as "embedded-only" so does not have its own datastore table.
15/12/02 11:05:18 INFO DataNucleus.Datastore: The class "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as "embedded-only" so does not have its own datastore table.
15/12/02 11:05:18 INFO metastore.ObjectStore: Initialized ObjectStore
15/12/02 11:05:19 WARN metastore.ObjectStore: Version information not found in metastore. hive.metastore.schema.verification is not enabled so recording the schema version 0.13.1aa
15/12/02 11:05:19 INFO metastore.HiveMetaStore: Added admin role in metastore
15/12/02 11:05:19 INFO metastore.HiveMetaStore: Added public role in metastore
15/12/02 11:05:19 INFO metastore.HiveMetaStore: No user is added in admin role, since config is empty
15/12/02 11:05:19 INFO session.SessionState: No Tez session required at this point. hive.execution.engine=mr.
15/12/02 11:05:19 INFO parse.ParseDriver: Parsing command: SELECT * FROM streamsummary
15/12/02 11:05:20 INFO parse.ParseDriver: Parse Completed
15/12/02 11:05:20 INFO hive.HiveContext: Initializing HiveMetastoreConnection version 0.13.1 using Spark classes.
15/12/02 11:05:20 INFO metastore.HiveMetaStore: 0: get_table : db=default tbl=streamsummary
15/12/02 11:05:20 INFO HiveMetaStore.audit: ugi=spark   ip=unknown-ip-addr  cmd=get_table : db=default tbl=streamsummary    
15/12/02 11:05:20 DEBUG myCompany.Main$: no such table streamsummary; line 1 pos 14

I basically run into the same 'no such table' problem for any time my application needs to read from or write to the Hive tables.

Posting hive-site.xml here for reference.

  <configuration>

    <property>
      <name>ambari.hive.db.schema.name</name>
      <value>hive</value>
    </property>

    <property>
      <name>datanucleus.autoCreateSchema</name>
      <value>false</value>
    </property>

    <property>
      <name>datanucleus.cache.level2.type</name>
      <value>none</value>
    </property>

    <property>
      <name>hive.auto.convert.join</name>
      <value>true</value>
    </property>

    <property>
      <name>hive.auto.convert.join.noconditionaltask</name>
      <value>true</value>
    </property>

    <property>
      <name>hive.auto.convert.join.noconditionaltask.size</name>
      <value>1968526677</value>
    </property>

    <property>
      <name>hive.auto.convert.sortmerge.join</name>
      <value>true</value>
    </property>

    <property>
      <name>hive.auto.convert.sortmerge.join.to.mapjoin</name>
      <value>false</value>
    </property>

    <property>
      <name>hive.cbo.enable</name>
      <value>true</value>
    </property>

    <property>
      <name>hive.cli.print.header</name>
      <value>false</value>
    </property>

    <property>
      <name>hive.cluster.delegation.token.store.class</name>
      <value>org.apache.hadoop.hive.thrift.ZooKeeperTokenStore</value>
    </property>

    <property>
      <name>hive.cluster.delegation.token.store.zookeeper.connectString</name>
      <value>worker1.xxx.com:2181,worker3.xxx.com:2181,worker2.xxx.com:2181</value>
    </property>

    <property>
      <name>hive.cluster.delegation.token.store.zookeeper.znode</name>
      <value>/hive/cluster/delegation</value>
    </property>

    <property>
      <name>hive.compactor.abortedtxn.threshold</name>
      <value>1000</value>
    </property>

    <property>
      <name>hive.compactor.check.interval</name>
      <value>300L</value>
    </property>

    <property>
      <name>hive.compactor.delta.num.threshold</name>
      <value>10</value>
    </property>

    <property>
      <name>hive.compactor.delta.pct.threshold</name>
      <value>0.1f</value>
    </property>

    <property>
      <name>hive.compactor.initiator.on</name>
      <value>true</value>
    </property>

    <property>
      <name>hive.compactor.worker.threads</name>
      <value>0</value>
    </property>

    <property>
      <name>hive.compactor.worker.timeout</name>
      <value>86400L</value>
    </property>

    <property>
      <name>hive.compute.query.using.stats</name>
      <value>true</value>
    </property>

    <property>
      <name>hive.conf.restricted.list</name>
      <value>hive.security.authenticator.manager,hive.security.authorization.manager,hive.users.in.admin.role</value>
    </property>

    <property>
      <name>hive.convert.join.bucket.mapjoin.tez</name>
      <value>false</value>
    </property>

    <property>
      <name>hive.default.fileformat</name>
      <value>TextFile</value>
    </property>

    <property>
      <name>hive.default.fileformat.managed</name>
      <value>TextFile</value>
    </property>

    <property>
      <name>hive.enforce.bucketing</name>
      <value>false</value>
    </property>

    <property>
      <name>hive.enforce.sorting</name>
      <value>true</value>
    </property>

    <property>
      <name>hive.enforce.sortmergebucketmapjoin</name>
      <value>true</value>
    </property>

    <property>
      <name>hive.exec.compress.intermediate</name>
      <value>false</value>
    </property>

    <property>
      <name>hive.exec.compress.output</name>
      <value>false</value>
    </property>

    <property>
      <name>hive.exec.dynamic.partition</name>
      <value>true</value>
    </property>

    <property>
      <name>hive.exec.dynamic.partition.mode</name>
      <value>nonstrict</value>
    </property>

    <property>
      <name>hive.exec.failure.hooks</name>
      <value>org.apache.hadoop.hive.ql.hooks.ATSHook</value>
    </property>

    <property>
      <name>hive.exec.max.created.files</name>
      <value>100000</value>
    </property>

    <property>
      <name>hive.exec.max.dynamic.partitions</name>
      <value>5000</value>
    </property>

    <property>
      <name>hive.exec.max.dynamic.partitions.pernode</name>
      <value>2000</value>
    </property>

    <property>
      <name>hive.exec.orc.compression.strategy</name>
      <value>SPEED</value>
    </property>

    <property>
      <name>hive.exec.orc.default.compress</name>
      <value>ZLIB</value>
    </property>

    <property>
      <name>hive.exec.orc.default.stripe.size</name>
      <value>67108864</value>
    </property>

    <property>
      <name>hive.exec.orc.encoding.strategy</name>
      <value>SPEED</value>
    </property>

    <property>
      <name>hive.exec.parallel</name>
      <value>false</value>
    </property>

    <property>
      <name>hive.exec.parallel.thread.number</name>
      <value>8</value>
    </property>

    <property>
      <name>hive.exec.post.hooks</name>
      <value>org.apache.hadoop.hive.ql.hooks.ATSHook</value>
    </property>

    <property>
      <name>hive.exec.pre.hooks</name>
      <value>org.apache.hadoop.hive.ql.hooks.ATSHook</value>
    </property>

    <property>
      <name>hive.exec.reducers.bytes.per.reducer</name>
      <value>67108864</value>
    </property>

    <property>
      <name>hive.exec.reducers.max</name>
      <value>1009</value>
    </property>

    <property>
      <name>hive.exec.scratchdir</name>
      <value>/tmp/hive</value>
    </property>

    <property>
      <name>hive.exec.submit.local.task.via.child</name>
      <value>true</value>
    </property>

    <property>
      <name>hive.exec.submitviachild</name>
      <value>false</value>
    </property>

    <property>
      <name>hive.execution.engine</name>
      <value>tez</value>
    </property>

    <property>
      <name>hive.fetch.task.aggr</name>
      <value>false</value>
    </property>

    <property>
      <name>hive.fetch.task.conversion</name>
      <value>more</value>
    </property>

    <property>
      <name>hive.fetch.task.conversion.threshold</name>
      <value>1073741824</value>
    </property>

    <property>
      <name>hive.limit.optimize.enable</name>
      <value>true</value>
    </property>

    <property>
      <name>hive.limit.pushdown.memory.usage</name>
      <value>0.04</value>
    </property>

    <property>
      <name>hive.map.aggr</name>
      <value>true</value>
    </property>

    <property>
      <name>hive.map.aggr.hash.force.flush.memory.threshold</name>
      <value>0.9</value>
    </property>

    <property>
      <name>hive.map.aggr.hash.min.reduction</name>
      <value>0.5</value>
    </property>

    <property>
      <name>hive.map.aggr.hash.percentmemory</name>
      <value>0.5</value>
    </property>

    <property>
      <name>hive.mapjoin.bucket.cache.size</name>
      <value>10000</value>
    </property>

    <property>
      <name>hive.mapjoin.optimized.hashtable</name>
      <value>true</value>
    </property>

    <property>
      <name>hive.mapred.reduce.tasks.speculative.execution</name>
      <value>false</value>
    </property>

    <property>
      <name>hive.merge.mapfiles</name>
      <value>true</value>
    </property>

    <property>
      <name>hive.merge.mapredfiles</name>
      <value>false</value>
    </property>

    <property>
      <name>hive.merge.orcfile.stripe.level</name>
      <value>true</value>
    </property>

    <property>
      <name>hive.merge.rcfile.block.level</name>
      <value>true</value>
    </property>

    <property>
      <name>hive.merge.size.per.task</name>
      <value>256000000</value>
    </property>

    <property>
      <name>hive.merge.smallfiles.avgsize</name>
      <value>16000000</value>
    </property>

    <property>
      <name>hive.merge.tezfiles</name>
      <value>false</value>
    </property>

    <property>
      <name>hive.metastore.authorization.storage.checks</name>
      <value>false</value>
    </property>

    <property>
      <name>hive.metastore.cache.pinobjtypes</name>
      <value>Table,Database,Type,FieldSchema,Order</value>
    </property>

    <property>
      <name>hive.metastore.client.connect.retry.delay</name>
      <value>5s</value>
    </property>

    <property>
      <name>hive.metastore.client.socket.timeout</name>
      <value>1800s</value>
    </property>

    <property>
      <name>hive.metastore.connect.retries</name>
      <value>24</value>
    </property>

    <property>
      <name>hive.metastore.execute.setugi</name>
      <value>true</value>
    </property>

    <property>
      <name>hive.metastore.failure.retries</name>
      <value>24</value>
    </property>

    <property>
      <name>hive.metastore.kerberos.keytab.file</name>
      <value>/etc/security/keytabs/hive.service.keytab</value>
    </property>

    <property>
      <name>hive.metastore.kerberos.principal</name>
      <value>hive/[email protected]</value>
    </property>

    <property>
      <name>hive.metastore.pre.event.listeners</name>
      <value>org.apache.hadoop.hive.ql.security.authorization.AuthorizationPreEventListener</value>
    </property>

    <property>
      <name>hive.metastore.sasl.enabled</name>
      <value>false</value>
    </property>

    <property>
      <name>hive.metastore.server.max.threads</name>
      <value>100000</value>
    </property>

    <property>
      <name>hive.metastore.uris</name>
      <value>thrift://host.xxx.com:9083</value>
    </property>

    <property>
      <name>hive.metastore.warehouse.dir</name>
      <value>/apps/hive/warehouse</value>
    </property>

    <property>
      <name>hive.optimize.bucketmapjoin</name>
      <value>true</value>
    </property>

    <property>
      <name>hive.optimize.bucketmapjoin.sortedmerge</name>
      <value>false</value>
    </property>

    <property>
      <name>hive.optimize.constant.propagation</name>
      <value>true</value>
    </property>

    <property>
      <name>hive.optimize.index.filter</name>
      <value>true</value>
    </property>

    <property>
      <name>hive.optimize.metadataonly</name>
      <value>true</value>
    </property>

    <property>
      <name>hive.optimize.null.scan</name>
      <value>true</value>
    </property>

    <property>
      <name>hive.optimize.reducededuplication</name>
      <value>true</value>
    </property>

    <property>
      <name>hive.optimize.reducededuplication.min.reducer</name>
      <value>4</value>
    </property>

    <property>
      <name>hive.optimize.sort.dynamic.partition</name>
      <value>false</value>
    </property>

    <property>
      <name>hive.orc.compute.splits.num.threads</name>
      <value>10</value>
    </property>

    <property>
      <name>hive.orc.splits.include.file.footer</name>
      <value>false</value>
    </property>

    <property>
      <name>hive.prewarm.enabled</name>
      <value>false</value>
    </property>

    <property>
      <name>hive.prewarm.numcontainers</name>
      <value>3</value>
    </property>

    <property>
      <name>hive.security.authenticator.manager</name>
      <value>org.apache.hadoop.hive.ql.security.ProxyUserAuthenticator</value>
    </property>

    <property>
      <name>hive.security.authorization.enabled</name>
      <value>false</value>
    </property>

    <property>
      <name>hive.security.authorization.manager</name>
      <value>org.apache.hadoop.hive.ql.security.authorization.plugin.sqlstd.SQLStdConfOnlyAuthorizerFactory</value>
    </property>

    <property>
      <name>hive.security.metastore.authenticator.manager</name>
      <value>org.apache.hadoop.hive.ql.security.HadoopDefaultMetastoreAuthenticator</value>
    </property>

    <property>
      <name>hive.security.metastore.authorization.auth.reads</name>
      <value>true</value>
    </property>

    <property>
      <name>hive.security.metastore.authorization.manager</name>
      <value>org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider</value>
    </property>

    <property>
      <name>hive.server2.allow.user.substitution</name>
      <value>true</value>
    </property>

    <property>
      <name>hive.server2.authentication</name>
      <value>NONE</value>
    </property>

    <property>
      <name>hive.server2.authentication.spnego.keytab</name>
      <value>HTTP/[email protected]</value>
    </property>

    <property>
      <name>hive.server2.authentication.spnego.principal</name>
      <value>/etc/security/keytabs/spnego.service.keytab</value>
    </property>

    <property>
      <name>hive.server2.enable.doAs</name>
      <value>false</value>
    </property>

    <property>
      <name>hive.server2.logging.operation.enabled</name>
      <value>true</value>
    </property>

    <property>
      <name>hive.server2.logging.operation.log.location</name>
      <value>${system:java.io.tmpdir}/${system:user.name}/operation_logs</value>
    </property>

    <property>
      <name>hive.server2.support.dynamic.service.discovery</name>
      <value>true</value>
    </property>

    <property>
      <name>hive.server2.table.type.mapping</name>
      <value>CLASSIC</value>
    </property>

    <property>
      <name>hive.server2.tez.default.queues</name>
      <value>default</value>
    </property>

    <property>
      <name>hive.server2.tez.initialize.default.sessions</name>
      <value>false</value>
    </property>

    <property>
      <name>hive.server2.tez.sessions.per.default.queue</name>
      <value>1</value>
    </property>

    <property>
      <name>hive.server2.thrift.http.path</name>
      <value>cliservice</value>
    </property>

    <property>
      <name>hive.server2.thrift.http.port</name>
      <value>10001</value>
    </property>

    <property>
      <name>hive.server2.thrift.max.worker.threads</name>
      <value>500</value>
    </property>

    <property>
      <name>hive.server2.thrift.port</name>
      <value>10000</value>
    </property>

    <property>
      <name>hive.server2.thrift.sasl.qop</name>
      <value>auth</value>
    </property>

    <property>
      <name>hive.server2.transport.mode</name>
      <value>binary</value>
    </property>

    <property>
      <name>hive.server2.use.SSL</name>
      <value>false</value>
    </property>

    <property>
      <name>hive.server2.zookeeper.namespace</name>
      <value>hiveserver2</value>
    </property>

    <property>
      <name>hive.smbjoin.cache.rows</name>
      <value>10000</value>
    </property>

    <property>
      <name>hive.stats.autogather</name>
      <value>true</value>
    </property>

    <property>
      <name>hive.stats.dbclass</name>
      <value>fs</value>
    </property>

    <property>
      <name>hive.stats.fetch.column.stats</name>
      <value>true</value>
    </property>

    <property>
      <name>hive.stats.fetch.partition.stats</name>
      <value>true</value>
    </property>

    <property>
      <name>hive.support.concurrency</name>
      <value>false</value>
    </property>

    <property>
      <name>hive.tez.auto.reducer.parallelism</name>
      <value>true</value>
    </property>

    <property>
      <name>hive.tez.container.size</name>
      <value>5632</value>
    </property>

    <property>
      <name>hive.tez.cpu.vcores</name>
      <value>-1</value>
    </property>

    <property>
      <name>hive.tez.dynamic.partition.pruning</name>
      <value>true</value>
    </property>

    <property>
      <name>hive.tez.dynamic.partition.pruning.max.data.size</name>
      <value>104857600</value>
    </property>

    <property>
      <name>hive.tez.dynamic.partition.pruning.max.event.size</name>
      <value>1048576</value>
    </property>

    <property>
      <name>hive.tez.input.format</name>
      <value>org.apache.hadoop.hive.ql.io.HiveInputFormat</value>
    </property>

    <property>
      <name>hive.tez.java.opts</name>
      <value>-server -Djava.net.preferIPv4Stack=true -XX:NewRatio=8 -XX:+UseNUMA -XX:+UseG1GC -XX:+ResizeTLAB -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps</value>
    </property>

    <property>
      <name>hive.tez.log.level</name>
      <value>INFO</value>
    </property>

    <property>
      <name>hive.tez.max.partition.factor</name>
      <value>2.0</value>
    </property>

    <property>
      <name>hive.tez.min.partition.factor</name>
      <value>0.25</value>
    </property>

    <property>
      <name>hive.tez.smb.number.waves</name>
      <value>0.5</value>
    </property>

    <property>
      <name>hive.txn.manager</name>
      <value>org.apache.hadoop.hive.ql.lockmgr.DummyTxnManager</value>
    </property>

    <property>
      <name>hive.txn.max.open.batch</name>
      <value>1000</value>
    </property>

    <property>
      <name>hive.txn.timeout</name>
      <value>300</value>
    </property>

    <property>
      <name>hive.user.install.directory</name>
      <value>/user/</value>
    </property>

    <property>
      <name>hive.vectorized.execution.enabled</name>
      <value>true</value>
    </property>

    <property>
      <name>hive.vectorized.execution.reduce.enabled</name>
      <value>false</value>
    </property>

    <property>
      <name>hive.vectorized.groupby.checkinterval</name>
      <value>4096</value>
    </property>

    <property>
      <name>hive.vectorized.groupby.flush.percent</name>
      <value>0.1</value>
    </property>

    <property>
      <name>hive.vectorized.groupby.maxentries</name>
      <value>100000</value>
    </property>

    <property>
      <name>hive.zookeeper.client.port</name>
      <value>2181</value>
    </property>

    <property>
      <name>hive.zookeeper.namespace</name>
      <value>hive_zookeeper_namespace</value>
    </property>

    <property>
      <name>hive.zookeeper.quorum</name>
      <value>worker1.xxx.com:2181,worker3.xxx.com:2181,worker2.xxx.com:2181</value>
    </property>

    <property>
      <name>javax.jdo.option.ConnectionDriverName</name>
      <value>com.mysql.jdbc.Driver</value>
    </property>

    <property>
      <name>javax.jdo.option.ConnectionPassword</name>
      <value>hive</value>
    </property>

    <property>
      <name>javax.jdo.option.ConnectionURL</name>
      <value>jdbc:mysql://host.xxx.com/hive?createDatabaseIfNotExist=true</value>
    </property>

    <property>
      <name>javax.jdo.option.ConnectionUserName</name>
      <value>hive</value>
    </property>

  </configuration>
like image 803
Ton Torres Avatar asked Dec 10 '22 19:12

Ton Torres

1 Answers

I posted this same question on the Hortonworks community, and I resolved the issue with the help of this answer.

The gist of it is this: when submitting the application, the --files argument has to come before the --jars argument, and the copy of hive-site.xml to use is the one in the Spark conf dir, not in $HIVE_HOME/conf/hive-site.xml. Hence:

  ./bin/spark-submit \
  --class com.myCompany.Main \
  --master yarn-cluster \
  --num-executors 3 \
  --driver-memory 1g \
  --executor-memory 11g \
  --executor-cores 1 \
  --files /usr/hdp/current/spark-client/conf/hive-site.xml \
  --jars lib/datanucleus-api-jdo-3.2.6.jar,lib/datanucleus-rdbms-3.2.9.jar,lib/datanucleus-core-3.2.10.jar \
  /home/spark/apps/YarnClusterTest.jar
like image 153
Ton Torres Avatar answered Dec 28 '22 08:12

Ton Torres
Sign in to Comment

Related questions

Pyspark: Using repartitionAndSortWithinPartitions with multiple sort Critiria
Why spark keeps on recomputing an RDD?
How to use CROSS JOIN and CROSS APPLY in Spark SQL
TypeError: 'Builder' object is not callable Spark structured streaming
EMR 5.x | Spark on Yarn | Exit code 137 and Java heap space Error
Spark dataframe select rows with at least one null or blank in any column of that row
Generic T as Spark Dataset[T] constructor
Spark UDAF with ArrayType as bufferSchema performance issues
How to use AWS Glue / Spark to convert CSVs partitioned and split in S3 to partitioned and split Parquet
How to extract all elements from array of structs?
How to check if key exists in spark sql map type
Spark Dataframe: Select distinct rows
Why "databricks-connect test" does not work after configurate Databricks Connect?
Which Scala version does Spark 2.4.3 uses?
having Spark process partitions concurrently, using a single dev/test machine
Provider org.apache.spark.sql.avro.AvroFileFormat could not be instantiated
spark delta overwrite a specific partition
How to create date from year, month and day in PySpark?
Scala with spark - "javax.servlet.ServletRegistration"'s signer information does not match signer information of other classes in the same package
How to work with Apache Spark using Intellij Idea?

Donate For Us

If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!