How do I uniquely identify computers visiting my web site?

Question

Introduction

I don't know if there is or ever will be a way to uniquely identify machines using a browser alone. The main reasons are:

• You will need to save data on the users computer. This data can be deleted by the user any time. Unless you have a way to recreate this data which is unique for each and every machine then your stuck.
• Validation. You need to guard against spoofing, session hijacking, etc.

Even if there are ways to track a computer without using cookies there will always be a way to bypass it and software that will do this automatically. If you really need to track something based on a computer you will have to write a native application (Apple Store / Android Store / Windows Program / etc).

I might not be able to give you an answer to the question you asked but I can show you how to implement session tracking. With session tracking you try to track the browsing session instead of the computer visiting your site. By tracking the session, your database schema will look like this:

sesssion:
  sessionID: string
  // Global session data goes here
  
  computers: [{
     BrowserID: string
     ComputerID: string
     FingerprintID: string
     userID: string
     authToken: string
     ipAddresses: ["203.525....", "203.525...", ...]
     // Computer session data goes here
  }, ...]

Advantages of session based tracking:

1. For logged in users, you can always generate the same session id from the users username / password / email.
2. You can still track guest users using sessionID.
3. Even if several people use the same computer (ie cybercafe) you can track them separately if they log in.

Disadvantages of session based tracking:

1. Sessions are browser based and not computer based. If a user uses 2 different browsers it will result in 2 different sessions. If this is a problem you can stop reading here.
2. Sessions expire if user is not logged in. If a user is not logged in, then they will use a guest session which will be invalidated if user deletes cookies and browser cache.

Implementation

There are many ways of implementing this. I don't think I can cover them all I'll just list my favorite which would make this an opinionated answer. Bear that in mind.

Basics

I will track the session by using what is known as a forever cookie. This is data which will automagically recreate itself even if the user deletes his cookies or updates his browser. It will not however survive the user deleting both their cookies and their browsing cache.

To implement this I will use the browsers caching mechanism (RFC), WebStorage API (MDN) and browser cookies (RFC, Google Analytics).

Legal

In order to utilize tracking ids you need to add them to both your privacy policy and your terms of use preferably under the sub-heading Tracking. We will use the following keys on both document.cookie and window.localStorage:

• _ga: Google Analytics data
• __utma: Google Analytics tracking cookie
• sid: SessionID

Make sure you include links to your Privacy policy and terms of use on all pages that use tracking.

Where do I store my session data?

You can either store your session data in your website database or on the users computer. Since I normally work on smaller sites (let than 10 thousand continuous connections) that use 3rd party applications (Google Analytics / Clicky / etc) it's best for me to store data on clients computer. This has the following advantages:

1. No database lookup / overhead / load / latency / space / etc.
2. User can delete their data whenever they want without the need to write me annoying emails.

and disadvantages:

1. Data has to be encrypted / decrypted and signed / verified which creates cpu overhead on client (not so bad) and server (bah!).
2. Data is deleted when user deletes their cookies and cache. (this is what I want really)
3. Data is unavailable for analytics when users go off-line. (analytics for currently browsing users only)

UUIDS

• BrowserID: Unique id generated from the browsers user agent string. Browser|BrowserVersion|OS|OSVersion|Processor|MozzilaMajorVersion|GeckoMajorVersion
• ComputerID: Generated from users IP Address and HTTPS session key. getISP(requestIP)|getHTTPSClientKey()
• FingerPrintID: JavaScript based fingerprinting based on a modified fingerprint.js. FingerPrint.get()
• SessionID: Random key generated when user 1st visits site. BrowserID|ComputerID|randombytes(256)
• GoogleID: Generated from __utma cookie. getCookie(__utma).uniqueid

Mechanism

The other day I was watching the wendy williams show with my girlfriend and was completely horrified when the host advised her viewers to delete their browser history at least once a month. Deleting browser history normally has the following effects:

1. Deletes history of visited websites.
2. Deletes cookies and window.localStorage (aww man).

Most modern browsers make this option readily available but fear not friends. For there is a solution. The browser has a caching mechanism to store scripts / images and other things. Usually even if we delete our history, this browser cache still remains. All we need is a way to store our data here. There are 2 methods of doing this. The better one is to use a SVG image and store our data inside its tags. This way data can still be extracted even if JavaScript is disabled using flash. However since that is a bit complicated I will demonstrate the other approach which uses JSONP (Wikipedia)

example.com/assets/js/tracking.js (actually tracking.php)

var now = new Date();
var window.__sid = "SessionID"; // Server generated

setCookie("sid", window.__sid, now.setFullYear(now.getFullYear() + 1, now.getMonth(), now.getDate() - 1));

if( "localStorage" in window ) {
  window.localStorage.setItem("sid", window.__sid);
}

Now we can get our session key any time:

window.__sid || window.localStorage.getItem("sid") || getCookie("sid") || ""

How do I make tracking.js stick in browser?

We can achieve this using Cache-Control, Last-Modified and ETag HTTP headers. We can use the SessionID as value for etag header:

setHeaders({
  "ETag": SessionID,
  "Last-Modified": new Date(0).toUTCString(),
  "Cache-Control": "private, max-age=31536000, s-max-age=31536000, must-revalidate"
})

Last-Modified header tells the browser that this file is basically never modified. Cache-Control tells proxies and gateways not to cache the document but tells the browser to cache it for 1 year.

The next time the browser requests the document, it will send If-Modified-Since and If-None-Match headers. We can use these to return a 304 Not Modified response.

example.com/assets/js/tracking.php

$sid = getHeader("If-None-Match") ?: getHeader("if-none-match") ?: getHeader("IF-NONE-MATCH") ?: ""; 
$ifModifiedSince = hasHeader("If-Modified-Since") ?: hasHeader("if-modified-since") ?: hasHeader("IF-MODIFIED-SINCE");

if( validateSession($sid) ) {
  if( sessionExists($sid) ) {
    continueSession($sid);
    send304();
  } else {
    startSession($sid);
    send304();
  }
} else if( $ifModifiedSince ) {
  send304();
} else {
  startSession();
  send200();
}

Now every time the browser requests tracking.js our server will respond with a 304 Not Modified result and force an execute of the local copy of tracking.js.

I still don't understand. Explain it to me

Lets suppose the user clears their browsing history and refreshes the page. The only thing left on the users computer is a copy of tracking.js in browser cache. When the browser requests tracking.js it recieves a 304 Not Modified response which causes it to execute the 1st version of tracking.js it recieved. tracking.js executes and restores the SessionID that was deleted.

Validation

Suppose Haxor X steals our customers cookies while they are still logged in. How do we protect them? Cryptography and Browser fingerprinting to the rescue. Remember our original definition for SessionID was:

BrowserID|ComputerID|randomBytes(256)

We can change this to:

Timestamp|BrowserID|ComputerID|encrypt(randomBytes(256), hk)|sign(Timestamp|BrowserID|ComputerID|randomBytes(256), hk)

Where hk = sign(Timestamp|BrowserID|ComputerID, serverKey).

Now we can validate our SessionID using the following algorithm:

if( getTimestamp($sid) is older than 1 year ) return false;
if( getBrowserID($sid) !== createBrowserID($_Request, $_Server) ) return false;
if( getComputerID($sid) !== createComputerID($_Request, $_Server) return false;

$hk = sign(getTimestamp($sid) + getBrowserID($sid) + getComputerID($sid), $SERVER["key"]);

if( !verify(getTimestamp($sid) + getBrowserID($sid) + getComputerID($sid) + decrypt(getRandomBytes($sid), hk), getSignature($sid), $hk) ) return false;

return true; 

Now in order for Haxor's attack to work they must:

1. Have same ComputerID. That means they have to have the same ISP provider as victim (Tricky). This will give our victim the opportunity to take legal action in their own country. Haxor must also obtain HTTPS session key from victim (Hard).
2. Have same BrowserID. Anyone can spoof User-Agent string (Annoying).
3. Be able to create their own fake SessionID (Very Hard). Volume atacks won't work because we use a time-stamp to generate encryption / signing key so basically its like generating a new key for each session. On top of that we encrypt random bytes so a simple dictionary attack is also out of the question.

We can improve validation by forwarding GoogleID and FingerprintID (via ajax or hidden fields) and matching against those.

if( GoogleID != getStoredGoodleID($sid) ) return false;
if( byte_difference(FingerPrintID, getStoredFingerprint($sid) > 10%) return false;

These people have developed a fingerprinting method for recognising a user with a high level of accuracy:

https://panopticlick.eff.org/static/browser-uniqueness.pdf

We investigate the degree to which modern web browsers are subject to “device fingerprinting” via the version and configuration information that they will transmit to websites upon request. We implemented one possible fingerprinting algorithm, and collected these fingerprints from a large sample of browsers that visited our test side, panopticlick.eff.org. We observe that the distribution of our finger- print contains at least 18.1 bits of entropy, meaning that if we pick a browser at random, at best we expect that only one in 286,777 other browsers will share its fingerprint. Among browsers that support Flash or Java, the situation is worse, with the average browser carrying at least 18.8 bits of identifying information. 94.2% of browsers with Flash or Java were unique in our sample.

By observing returning visitors, we estimate how rapidly browser fingerprints might change over time. In our sample, fingerprints changed quite rapidly, but even a simple heuristic was usually able to guess when a fingerprint was an “upgraded” version of a previously observed browser’s fingerprint, with 99.1% of guesses correct and a false positive rate of only 0.86%.

We discuss what privacy threat browser fingerprinting poses in practice, and what countermeasures may be appropriate to prevent it. There is a tradeoff between protection against fingerprintability and certain kinds of debuggability, which in current browsers is weighted heavily against privacy. Paradoxically, anti-fingerprinting privacy technologies can be self- defeating if they are not used by a sufficient number of people; we show that some privacy measures currently fall victim to this paradox, but others do not...

It's not possible to identify the computers accessing a web site without the cooperation of their owners. If they let you, however, you can store a cookie to identify the machine when it visits your site again. The key is, the visitor is in control; they can remove the cookie and appear as a new visitor any time they wish.

A possibility is using flash cookies:

• Ubiquitous availability (95 percent of visitors will probably have flash)
• You can store more data per cookie (up to 100 KB)
• Shared across browsers, so more likely to uniquely identify a machine
• Clearing the browser cookies does not remove the flash cookies.

You'll need to build a small (hidden) flash movie to read and write them.

Whatever route you pick, make sure your users opt IN to being tracked, otherwise you're invading their privacy and become one of the bad guys.

You may want to try setting a unique ID in an evercookie (it will work cross browser, see their FAQs): http://samy.pl/evercookie/

There is also a company called ThreatMetrix that is used by a lot of big companies to solve this problem: http://threatmetrix.com/our-solutions/solutions-by-product/trustdefender-id/ They are quite expensive and some of their other products aren't very good, but their device id works well.

Finally, there is this open source jquery implementation of the panopticlick idea: https://github.com/carlo/jquery-browser-fingerprint It looks pretty half baked right now but could be expanded upon.

Hope it helps!