In what cases HTTP referer will be truncated

Question

I'm trying to understand the behavior of HTTP referer header. I noticed that sometimes the referer is full (full URL, including path and query string) but mostly it includes the domain only.

For example 'https://www.google.com/' instead of 'https://www.google.com/search?q=http+referer+truncated&oq=http+referer+truncated&aqs=chrome..69i57.6485j0j1&sourceid=chrome&ie=UTF-8#q=http+referer+is+not+full'

Are there any rules as to when the refere and is full and when it's truncated?

Answer 1

Updated details as of Nov 2020...

Many browsers have started to default to a stricter referrer policy (strict-origin-when-cross-origin) when making a cross-domain request instead of the old default (no-referrer-when-downgrade). This will most often result in truncated urls, but occasionally means that the referrer will not be set at all (no-referrer).

Here is an excerpt from a good article about this: https://plausible.io/blog/referrer-policy

Chrome is using strict-origin-when-cross-origin from version 85. Strict-origin-when-cross-origin is where the full path is sent if on the same domain but only sends the domain itself if going to another domain. Previously it used no-referrer-when-downgrade.

Firefox is using no-referrer-when-downgrade by default. It always passes the full path unless the request is sent from HTTPS to HTTP. Firefox is using strict-origin-when-cross-origin in the Private Browsing tabs and for known trackers.

Edge is using no-referrer-when-downgrade. Same as Firefox.

Safari is using strict-origin-when-cross-origin. Same as Chrome.

Brave is using no-referrer where the referrer header is completely removed. It never shares the full URL even for same-origin requests and you cannot even see the domain name for cross-origin requests.

Answer 2

HTTP referrer headers are created by browsers according to desired criteria using Referrer Policy even though there is a general standard used by majority of the browsers there are some differences about how the browsers handles the servers instructions, mainly mobile web browsers are the ones which does not cooperate nicely with WWWC recommendations on this matter.

So why is there need for different HTTP referrer headers? To understand this we need to look at what are these headers are used for first. Main purpose in its simplest form is "carrying information from the originating page to the new page".

Everywhere we see the word "information" in the web there is a information security concept attached to it and HTTP header is no different. Depending on what kind of information headers carry, server can specify the type of referrer policy needs to be used. Here is the list of referrer policies from W3

enum ReferrerPolicy { "", "no-referrer", "no-referrer-when-downgrade", "same-origin", "origin", "strict-origin", "origin-when-cross-origin", "strict-origin-when-cross-origin", "unsafe-url" };

Detailed information about each of these are available in the Referrer policy link i included above.

To give an example; Using google searching for "Yellow Pages". in this case

origin:https://www.google.ie

referer:https://www.google.ie/

Referrer Policy:origin

generated URL:https://www.google.ie/gen_204?atyp=i&ct=&cad=udla=3&ei=x65kGDkdyKGHDkF0KeoBg&e=12&zx=1494785478502

link to the first result is

https://www.google.ie/url?sa=t&rct=j&q=&esrc=s&source=web&cd=1&cad=rja&uact=8&ved=0ahUKEwiA26TfiHSGDFHKFAKHQXoCWUQFggoMBB&url=https%3A%2F%2Fwww.goldenpages.ie%2F&usg=AFQjCNGTG-tsBSFHgMkXw_GuvOcLEOD2hg

While the actual URL is https://www.goldenpages.ie/

When we actually click the link referrer changes to

Referer:https://www.goldenpages.ie/ and the referrer policy is

Referrer Policy:no-referrer-when-downgrade

This means if we click another link from the current page we won't see all the additional parameters similar to the ones we saw in the URL from google search results page.

To prove this is the case; click any link from the current page and watch the referrer header changing according to the the policy type (Which can be found in the associated js file if you use developer tools and inspect the network activity)

When i click the "List your business" link referrer stay as

https://www.goldenpages.ie/list-your-business/

and no other parameters are passed

So just to tidy up this messy explanation; What URL gets generated is dependent on what rules are set regarding to Referrer policy may that be a simple base rule with no parameters or a very long URL with loads of information relating to the user and origin of the navigation.

Note: URLs wont work i have jumbled some letters.