Presto Nodes with too much load

Question

I'm performing some queries over a tpch 100gb dataset on presto, I have 4 nodes, 1 master, 3 workers. When I try to run some queries, not all of them, I see on Presto web interface that the nodes die during the execution, resulting in query failure, the error is the following:

.facebook.presto.operator.PageTransportTimeoutException: Encountered too many errors talking to a worker node. The node may have crashed or been under too much load. This is probably a transient issue, so please retry your query in a few minutes.

I rebooted all nodes and presto service but the error remains, this problem doesn't exist if I run the same queries over a smaller dataset.Can someone provide some help on this problem?

Thanks

Answer 1

3 possible causes for this kind of error. You may ssh into one of worker to find out what the problem is when the query is running.

• High CPU

  Tune down the task.concurrency to, for example, 8

• High memory

  In the jvm.config, -Xmx should no more than 80% total memory. In the config.properties, query.max-memory-per-node should be no more than the half of Xmx number.

• Low open file limit

  Set in the /etc/security/limits.conf a larger number for the Presto process. The default is definitely way too low.