Menu
I'm using Apache Spark's spark-shell with mesos and zookeeper, which seems to working well except that I'm getting more logging from zookeeper than I'd like, which is a bit distracting:
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 1.5.0
/_/
Using Scala version 2.10.4 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_60)
Type in expressions to have them evaluated.
Type :help for more information.
2015-03-26 20:08:22,115:15978(0x7f83a1fbc740):ZOO_INFO@log_env@712: Client environment:zookeeper.version=zookeeper C client 3.4.6
2015-03-26 20:08:22,115:15978(0x7f83a1fbc740):ZOO_INFO@log_env@716: Client environment:host.name=myhost
2015-03-26 20:08:22,115:15978(0x7f83a1fbc740):ZOO_INFO@log_env@723: Client environment:os.name=Linux
2015-03-26 20:08:22,115:15978(0x7f83a1fbc740):ZOO_INFO@log_env@724: Client environment:os.arch=3.2.0-34-generic
2015-03-26 20:08:22,115:15978(0x7f83a1fbc740):ZOO_INFO@log_env@725: Client environment:os.version=#53-Ubuntu SMP Thu Nov 15 10:48:16 UTC 2012
2015-03-26 20:08:22,115:15978(0x7f83a1fbc740):ZOO_INFO@log_env@733: Client environment:user.name=myname
2015-03-26 20:08:22,115:15978(0x7f83a1fbc740):ZOO_INFO@zookeeper_init@786: Initiating client connection, host=localhost:9999 sessionTimeout=10000 watcher=0xffffffffff sessionId=0 sessionPasswd=<null> context=0x777777777777 flags=0
2015-03-26 20:08:22,115:15978(0x7f83a1fbc740):ZOO_INFO@check_events@1703: initiated connection to server [127.0.0.1:9999]
2015-03-26 20:08:22,115:15978(0x7f83a1fbc740):ZOO_INFO@check_events@1750: session establishment complete on server [127.0.0.1:9999], sessionId=0x11111111111111, negotiated timeout=10000
Spark context available as sc.
SQL context available as sqlContext.
scala>
I've tried changing $SPARK_CONF_DIR/log4j.properties
log4j.logger.org.apache.zookeeper=WARN
Which didn't work, but I'm not surprised since the mesos documentation mentions that the zookeeper logging goes directly to stderr:
NOTE: 3rd party log messages (e.g. ZooKeeper) are only written to stderr!
Adding a jar to spark's classpath that has a static block with a foreign-function interface call to the zookeeper C api function to change the log level seems like it might work, but I'm not sure, and I don't have any experience with calling C code from the JVM.
655
To change log levels as a root user, perform the following: To enable debug logging, run the following command: /subsystem=logging/root-logger=ROOT:change-root-log-level(level=DEBUG) To disable debug logging, run the following command: /subsystem=logging/root-logger=ROOT:change-root-log-level(level=INFO)
ZooKeeper service. Alter EXTRA_ARGS for the Kafka service variable to disable GC Logging. Alter EXTRA_ARGS for the ZooKeeper service variable to disable GC Logging.
For external ZooKeeper, the location of the log files is specified by the ZOO_LOG_DIR environment variable. By default, this directory is the same as the directory where ZooKeeper is launched.
ZooKeeper stores its data in a data directory and its transaction log in a transaction log directory. By default these two directories are the same. The server can (and should) be configured to store the transaction log files in a separate directory than the data files.
In addition to,
log4j.logger.org.apache.zookeeper=WARN
also configure the stderr log-level to print only logs with priority ERROR (and above).
# Send WARN or higher to stderr
log4j.appender.stderr = org.apache.log4j.ConsoleAppender
log4j.appender.stderr.Threshold = ERROR
log4j.appender.stderr.Target = System.err
Also a LevelRangeFilter might also prove useful as shown here.
173
Couldn't find anything on this either, but for JNI (Java-to-C) can offer this. Looks like neither zookeeper not mesos have web service or shells (zookeeper has zkshell though, but not usable in our case) that can be used.
27
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
