How to deal with sporadic BadStatusLine, CannotSendRequest errors in python WebDriver

Question

Since we started running selenium UI tests in jenkins, we noticed a small but annoying frequency of errors during tests. We get BadStatusLine and CannotSendRequest errors on seemingly random selenium actions (click, quit, visit, etc.).

They would usually look something like:

  File "/usr/lib/python2.7/unittest/case.py", line 327, in run
    testMethod()
  File "/home/jenkins/workspace/Create and Upload Functional Testing/shapeways/test_suite/Portal/CreateAndUpload/TestUploadWhenNotLoggedIn_ExpectLoginModal.py", line 22, in runTest
    self.dw.visit(ShapewaysUrlBuilder.build_model_upload_url())
  File "/home/jenkins/workspace/Create and Upload Functional Testing/webdriver/webdriverwrapper/WebDriverWrapper.py", line 212, in visit
    return self.execute_and_handle_webdriver_exceptions(lambda: _visit(url))
  File "/home/jenkins/workspace/Create and Upload Functional Testing/webdriver/webdriverwrapper/WebDriverWrapper.py", line 887, in execute_and_handle_webdriver_exceptions
    return function_to_execute()
  File "/home/jenkins/workspace/Create and Upload Functional Testing/webdriver/webdriverwrapper/WebDriverWrapper.py", line 212, in <lambda>
    return self.execute_and_handle_webdriver_exceptions(lambda: _visit(url))
  File "/home/jenkins/workspace/Create and Upload Functional Testing/webdriver/webdriverwrapper/WebDriverWrapper.py", line 205, in _visit
    return self.driver.get(url)
  File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/webdriver.py", line 185, in get
    self.execute(Command.GET, {'url': url})
  File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/webdriver.py", line 171, in execute
    response = self.command_executor.execute(driver_command, params)
  File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/remote_connection.py", line 349, in execute
    return self._request(command_info[0], url, body=data)
  File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/remote_connection.py", line 380, in _request
    resp = self._conn.getresponse()
  File "/usr/lib/python2.7/httplib.py", line 1030, in getresponse
    response.begin()
  File "/usr/lib/python2.7/httplib.py", line 407, in begin
    version, status, reason = self._read_status()
  File "/usr/lib/python2.7/httplib.py", line 371, in _read_status
    raise BadStatusLine(line)

This particular case came from the following stack:

• selenium==2.44.0
• python==2.7.3
• firefox==34.0
• jenkins
• xvfb (using the jenkins plugin for headless displays)

though we've seen these errors popping up all the time across many different version permutations of firefox/selenium.

I ran a tcpdump to capture the actual request sent right before the BadStatusLine error came up and got the following.

    POST /hub/session/ab64574a-4a17-447a-b2e8-5b0f5ed5e923/url HTTP/1.1 
    Host: 127.0.0.1:41246
    Accept-Encoding: identity Content-Length: 102
    Connection: keep-alive
    Content-type: application/json;charset="UTF-8"
    POST: /hub/session/ab64574a-4a17-447a-b2e8-5b0f5ed5e923/url
    Accept: application/json
    User-Agent: Python http auth

    {"url": "http://example.com/login", "sessionId": "ab64574a-4a17-447a-b2e8-5b0f5ed5e923"}

Response comes back with 0 bytes. So the BadStatusLine was caused by an empty response, which makes sense.

The question is, why would selenium's server return an empty response. If the server died, wouldn't we get a ConnectionError or something along those lines?

Answer 1

For a while, I had no repro and no idea what the cause was. I was finally able to repro by running:

import requests
import json

while True:

  requests.post('http://127.0.0.1/hub/session/', data=json.dumps({"url": "http://example.com/login", "sessionId": "ab64574a-4a17-447a-b2e8-5b0f5ed5e923"}))

While this was running, I quit the browser and got a BadStatusLine error! When I tried making that request again, that's when I got the expected "ConnectionError" that you would see from any dead server.

SO, what I suspect happens is that when the browser is sent the kill signal, there is a short window during its shutdown where any response will still be returned but with 0 bytes. That's why you get different types of exceptions for essentially the same problem (browser dies). Turns out we had a cron which was killing our browsers in the background.