ElasticSearch query hangs in laravel job queue

Question

I am currently using the laravel-elasticsearch provider for my ES queries. I am using this from within a job that is processed by my laravel queue(using beanstalkd). The problem I am having is that in long running jobs, eventually I am no longer able to insert data into elasticsearch. The job just hangs (no exceptions are thrown) I have narrowed it down to the code where I am making an ES call. Is it possible that my connection could become stale some how and not reconnect? my other thought is that it has something to do with using a Facade and it being a singleton.

here is what I am doing, this is not my exact code. but the code works just fine when it is not run in a long running job.I just wanted to provide some context.it inserts just fine, there are no issues with the code functioning until it is run after a long running process.

UPDATE:

I have narrowed the issue down to the elasticsearch-php library persisting a connection. I have my ES behind a load balancer that times out TCP connections after 5 min. the issue is that there is no keep alive in the ES php library. after 5 min the connection is closed but it does not close the connection on my end. is there a way to set a keep alive for elasticsearch-php? or a call to reset the connection?

//run some functions....
$params = array();
$params['body']  = array('somefield' => 'some data');
$params['index'] = 'my_index';
$params['type']  = 'my_type';
$params['id']    = 'my_id';
$ret = Es::update($params);//this is working just fine

//long running function here
$newparams = array();
$newparams['body']  = array('somefield' => 'some data');
$newparams['index'] = 'my_index';
$newparams['type']  = 'my_type';
$newparams['id']    = 'my_id';
$return = Es::update($newparams);//this will just hang

Answer 1

Gather around, and let's talk about long lasting queue processes.

Laravel queues work, with workers. Those rentless elves fetch the jobs from our queue service and processes them in a timely fashion.

However, after long periods of incessant work, our elves start to get tired and create errors randomly. Types of errors include failure to send mail using SwiftMailer, to process database transactions or connect to SSL ports.

What our elves need, is an energy drink, in the form of queue:restart, every 20 mins.

Steps to implement:

1) If you are on linux, type in console sudo crontab -e

2) Using vim, enter the following lines 0,20,40 * * * * cd /path/to/my/laravel/installation && php artisan queue:restart --env=yourenvironment 1>> /dev/null 2>&1

Replace /path/to/my/laravel/installation with your laravel's path.

And yourenvironment with your laravel's environment name.

Source: I have an elasticsearch installation with uptime of one month, and laravel polling and updating data in real-time using the shift31's package.

Documentation: http://laravel.com/docs/5.0/queues#daemon-queue-worker

Answer 2

What I ended up doing to fix this issue was add the following setting when creating the ES client.

'guzzleOptions' => array(
              'curl.options' => [
                  CURLOPT_FORBID_REUSE => true
              ]
          ),

hope this helps someone out.