Is server-side user agent detection/sniffing bad?

Question

Client-side user agent detection is known to be bad and discouraged in favor of feature detection. However, is it also bad to react differently based on the incoming user agent field in a HTTP request?

An example would be sending smaller or larger images based on whether the incoming user agent is mobile or desktop.

Answer 1

I think it depends what your motivation is. For example, in the mobile web sector what you are attempting to do is provide the user with something that looks sensible on their platform. Why be concerned about what user-agent the user is reporting, when it is purely for their own benefit? If they go to the effort of tricking you with a different user-agent, then they are the only person that suffers. The main trouble of course is false positives; it's not entirely reliable.

I follow the argument that you should not rely on it as such, but mobile developers are under attack from generic broad statements like this. Yes there are good alternatives, but across every browser you can imagine, this information can actually be useful at some point as the certainty begins to degrade.

What you certainly don't ever do with any plain-text header is use it to facilitate access control.

User agent detection is considered bad when there are better alternatives, but there is certainly no harm in including it in a detection process which degrades gracefully in certainty.

The issue I have with the whole process is that we are caught up in providing the user something sensible, but never seem to think it's acceptable to ask when you are uncertain. If you are uncertain about the user-agent, why not ask once and store? You can use the user-agent as a guideline.

So to conclude my thoughts, essentially the user-agent header is unreliable, so it is bad to rely on it. This doesn't mean you can't extract a degree of valuable information from it where more reliable options leave you in an uncertain state. In general it's wrong to conclude that it is bad. It's simply what you do with this information that makes it bad or not.

Update

After seeing your updates to the question, I have the following comments to contribute. Do I want to be sniffing image requests and providing the client with an image based on user agent?

If this is the only variable then maybe it could work, but it's rarely the case that the only thing you are varying is the images. I don't want to detect per request because I want to serve the client a coherent solution. This means I served them a page that causes them to request the correct resources. This page yields a single coherent solution for all of the integrated resources. All variations in this document work together for a particular view.

I respect that the chance of the user-agent string changing mid-view is so slim it doesn't seem worth worrying about. However adopting this principle also reduces the number of times you need to perform browser/platform detection, which can only be beneficial. This allows you to switch views on the client much more easily. If the client says actually you got the view wrong, I am a tablet not a phone, how do you go about correcting that? You serve the user a better page, otherwise you will need to be spoofing headers for your image requests... terrible idea. Don't use the user-agent string to serve generic resources like images.

Potential improvements

Platform identification is a very active area of modern developments in the web. As computing becomes more ubiquitous and platforms vary much more widely, our need to understand the platforms we are serving increases. I think the general solution to this problem under the current conditions is going to fall on fingerprinting and statistical analysis.

Consider this application - akinator.com - Notice how the statistical analysis from a huge set of sparse data is annoyingly accurate. In a limited environment (the set of browser configurations), you can imagine that we could ask the client's browser some questions. We then perform a statistical analysis on the response in some n-dimensional feature space. Using the user-agent as a dimension of this space is going to be useful and self limiting, depending on the results that you find. If it's largely inaccurate then it will see a large spread, and the amount of worth you derive from it will be self limiting.

Of course your ability to derive any value from this statistical model requires you to be able to obtain some verified truths. This could be, for example, running a JavaScript test-suite to detect client side js capabilities, or indeed, in uncertainty, you can actually ask the user to tell you what their platform is.

---

For further reading I'd refer you to this article by Mozilla

https://developer.mozilla.org/en/Browser_detection_using_the_user_agent

Today, looking for these strings are the only way to know that the device runs on a mobile device (resp. a tablet) before serving the HTML.

Answer 2

It depends. Using the user agent as the sole signal to branch the logic of your server-level code is dubious at best and insecure at worst, but it works for defining the rote capabilities of particular classes of browser and serving content to match their needs when the vanilla agent is supplied.

The scenario you've sketched out is a perfect illustration of this. Attempting to detect mobile browsers and downscale the content you send to them at the server level is entirely appropriate, because you're trying to adapt the user experience to fit their needs better (for example, by providing smaller images and better content flow to fit within the constraints of a smaller screen) while balancing them with the needs of your server (sending smaller images, thus generating less load and less bandwidth over the line). This strategy just needs refinement.

There are a few design principles you should always follow here to ensure your practice of user agent detection isn't seen as dubious by your users:

• Always provide the ability to view the full version of your site and plan your load profile accordingly. Otherwise, you will have people attempt to circumvent this by changing their agent.

• Always clearly define the modifications of your site content when you create a modal view. This will clear up any FUD surrounding the changes you may or may not have made.

• Always provide paths to the alternate versions of your site. For example, use something like http://mobile.example.org for migrating people to the mobile version, making the design-level assumption that when this path is requested, it's been explicitly asked for by your audience.

• Reward users for providing their correct agent credentials to you, by offering a better experience for them in terms of content and performance. Users will be happier when you've anticipated their needs and given them snappier performance on the version of the site they're browsing.

• Avoid abuse and manual redirection patterns. For example, don't block them with a big horking flyout advertisement for your mobile app when you detect they're running iOS. (Admittedly, this is a pet peeve of mine.)

• Never restrict access to areas of the site on a user agent basis (opting instead to sternly warn users about what won't work if they go off the rails and drafting your support policy around it). For example, many of us fondly remember changing our agents for sites "that work best in Internet Explorer," disallowing all other browsers. You shouldn't become one more example of this bad practice if it can be avoided.

In short: providing the correct user agent is a decision by the user. You can use this to define a default experience for users choosing to run their clients plain vanilla or ones that don't know any better. The goal here is to reward your users with not providing a false user agent, by giving them the options they need and the experience they desire while balancing their needs with your own. Anything beyond that will cause them to balk, and as such, should be considered extremely dubious.

You can certainly try to detect the browser by other means, but this is still an area of open research. Browser capabilities and fingerprints change as they compete on features, and attempting to play catch-up to optimize performance is often, currently, intractable.

I concur with this answer on the use of statistical analysis, so don't take me wrong here. But, as someone that actively works in this area, I can tell you there's no magic bullet that will give you perfect classification certainty. Heuristics, however, can and will help you balance load more effectively, and to that end, browser interrogation strategies can and do have use to you once you've clearly defined an acceptable rate for error.

Answer 3

In "standard-browser" scenario it is not bad BUT it is not reliable since a lot of browsers offer the user some config option/plugin/whatever to modify the user-agent.

In such a situation I would implement something similar to facebook - they detect based on UA (and possibly other things a.k.a. as "fingerprint analysis") whether to redirect to a mobbile version (i.e. http://m.facebook.com/...) or not (i.e. http://www.facebook.com...). At the same time they offer a URL param m2w which overrides this redirection mechanism.

Depending on the mobile carrier it might even happen that they have some content-aware proxy/cache which does scale/recompress images on-the-fly and appears on your end as a "normal" borwser...

Thinking of scenarios outside the browser... for example if you are serving some specific protocol (like WebDAV) this might be the only option to have some sort of "platform-specific" behaviour (for example the difference between OS X and Windows).