How to make chromedriver undetectable

Question

This is my first Stack Overflow question so please bear with me.

I have read this SO question, which lead me to wondering, is it possible to make chromedriver completely undetectable?

For my own curiosity's sake I have tested the method described and found that it was unsuccessful in creating a completely anonymous browser.

I read through the driver's documentation and found this:

partial interface Navigator { readonly attribute boolean webdriver; };

The webdriver IDL attribute of the Navigator interface must return the value of the webdriver-active flag, which is initially false.

This property allows websites to determine that the user agent is under control by WebDriver, and can be used to help mitigate denial-of-service attacks.

However, I cannot find where these tags are even located through the browser console or in the source code.

I would imagine this is responsible for the detection of chromedriver, however, after combing through the source code, I could not find this interface. As a result, it has left me wondering whether or not this feature is included in the current chromedriver. If not, I still know that the current chromedriver is detectable by websites and other services such as distill.

Answer 1

In order to use ChromeDriver undetectable to Distil checkpoints (which are described nicely in this stackoverflow post), you will need to ensure that your browser does not contain any variable in its window or document prototypes that reveal that you are using a webdriver, as the one you mention.

You can use software as Selenium along with ChromeDriver and Chrome as long as you take some precautions and make some fixes to the binaries. This method will apply only to headed version, if you wish to use headless, you would need to take additional measurements to pass window/rendering tests, described here.

---

1. Fix Chrome binary, or use an old version

First, lets deal with that navigator.webdriver set to True. This is defined by W3C protocol here as part of 'NavigatorAutomationInformation' of browsers, which extends the Navigator interface. How to remove it? That project has a lot of files, third party stuff, blink web runtimes, etc. So, instead of becoming crazy trying to figure out how this works, as Chromium is open-source, just be clever and search google for the commit which incorporated that. Here is the link. Pay attention to these files:

• third_party/WebKit/Source/core/frame/Navigator.h, which holds the line of code:

   `bool webdriver() const { return true; }` 
  
  This method is supossed to always return true, as you can see.
  

• third_party/WebKit/Source/core/frame/Navigator.idl, which contains the extensions of Navigators, included our

  `Navigator implements NavigatorAutomationInformation;` 
  
  which is being commited. Interesting, isn't it?
  

• third_party/WebKit/Source/core/frame/NavigatorAutomationInformation.idl contains the extension itself, with a read-only variable, which is webdriver:

       `[
           NoInterfaceObject, // Always used on target of 'implements'
           Exposed=(Window),
           RuntimeEnabled=AutomationControlled
       ] interface NavigatorAutomationInformation {
           readonly attribute boolean webdriver;
       };`
  

To get rid of this functionality, it should be enough commenting the line in Navigator.idl which extends Navigator with this functionality, and compile the source (compiling in linux here). However, this is a laborious task for almost any computer and can take several hours. If you look the date of the commit, it was on Oct 2017, so an option is to download any version of Chrome released before that date. To search for mirrors, you can google for inurl:/deb/pool/main/g/google-chrome-stable/.

---

2. Fix ChromeDriver

Distil checks the regex rule '/\$[a-z]dc_/' against window variables, and ChromeDriver adds one as mentioned here which satisfies that condition. As they mention, you have to edit call_function.js amongst the source code, and redefine the variable var key = '$cdc_asdjflasutopfhvcZLmcfl_';. with something else. Also, probably easier, you can use an hex editor to update the existing binary.

If you decided to use an older version of Chrome -I guess you did-, you will need to use an appropiate version of ChromeDriver. You can know which one is fine for your Chrome version in the ChromeDriver downloads webpage. For example, for Chrome v61 (which fits your needs), you could use ChromeDriver 2.34. Once done, just put the ChromeDriver binary on '/usr/bin/local'.

---

3. Take other precautions

• Pay attention to your user-agent.
• Don't perform too many repeated requests.
• Use a (random) delay between requests.
• Use the Chrome arguments used here to mimic a normal user profile.

Answer 2

You can't use Selenium's WebDriver itself to change UserAgent, which sounds like what you're really trying to do here.

However, that doesn't mean it can't be changed.

Enter PhantomJS.

Check out this answer. You can use that to disguise Selenium as a different browser, or pretty much anything else. Of course, if a website is determined to figure you out, there are plenty of clues that Selenium leaves behind (like clicking with perfect precision).