Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

In what cases HTTP referer will be truncated

I'm trying to understand the behavior of HTTP referer header. I noticed that sometimes the referer is full (full URL, including path and query string) but mostly it includes the domain only.

For example 'https://www.google.com/' instead of 'https://www.google.com/search?q=http+referer+truncated&oq=http+referer+truncated&aqs=chrome..69i57.6485j0j1&sourceid=chrome&ie=UTF-8#q=http+referer+is+not+full'

Are there any rules as to when the refere and is full and when it's truncated?

like image 893
danieln Avatar asked Jul 10 '17 11:07

danieln


People also ask

Why is HTTP referer empty?

There might be several reasons why the referer URL would be blank. It will/may be empty when the enduser: entered the site URL in browser address bar itself. visited the site by a browser-maintained bookmark.

Can http referer be spoofed?

In HTTP networking, typically on the World Wide Web, referer spoofing (based on a canonised misspelling of "referrer") sends incorrect referer information in an HTTP request in order to prevent a website from obtaining accurate data on the identity of the web page previously visited by the user.

How do you test a referrer policy?

Check If Referrer-Policy Is Enabled They aren't automatic, though they may have been included in web-apps you've installed (WordPress, Joomla, etc). A quick way to check is to go to www.securityheaders.io and do a scan of your website.

Is Referer header reliable?

Using HTTP_REFERER isn't reliable, its value is dependent on the HTTP Referer header sent by the browser or client application to the server and therefore can't be trusted because it can be manipulated.


2 Answers

Updated details as of Nov 2020...

Many browsers have started to default to a stricter referrer policy (strict-origin-when-cross-origin) when making a cross-domain request instead of the old default (no-referrer-when-downgrade). This will most often result in truncated urls, but occasionally means that the referrer will not be set at all (no-referrer).

Here is an excerpt from a good article about this: https://plausible.io/blog/referrer-policy

Chrome is using strict-origin-when-cross-origin from version 85. Strict-origin-when-cross-origin is where the full path is sent if on the same domain but only sends the domain itself if going to another domain. Previously it used no-referrer-when-downgrade.

Firefox is using no-referrer-when-downgrade by default. It always passes the full path unless the request is sent from HTTPS to HTTP. Firefox is using strict-origin-when-cross-origin in the Private Browsing tabs and for known trackers.

Edge is using no-referrer-when-downgrade. Same as Firefox.

Safari is using strict-origin-when-cross-origin. Same as Chrome.

Brave is using no-referrer where the referrer header is completely removed. It never shares the full URL even for same-origin requests and you cannot even see the domain name for cross-origin requests.

like image 87
seebigs Avatar answered Oct 04 '22 05:10

seebigs


HTTP referrer headers are created by browsers according to desired criteria using Referrer Policy even though there is a general standard used by majority of the browsers there are some differences about how the browsers handles the servers instructions, mainly mobile web browsers are the ones which does not cooperate nicely with WWWC recommendations on this matter.

So why is there need for different HTTP referrer headers? To understand this we need to look at what are these headers are used for first. Main purpose in its simplest form is "carrying information from the originating page to the new page".

Everywhere we see the word "information" in the web there is a information security concept attached to it and HTTP header is no different. Depending on what kind of information headers carry, server can specify the type of referrer policy needs to be used. Here is the list of referrer policies from W3

enum ReferrerPolicy { "", "no-referrer", "no-referrer-when-downgrade", "same-origin", "origin", "strict-origin", "origin-when-cross-origin", "strict-origin-when-cross-origin", "unsafe-url" };

Detailed information about each of these are available in the Referrer policy link i included above.

To give an example; Using google searching for "Yellow Pages". in this case

origin:https://www.google.ie

referer:https://www.google.ie/

Referrer Policy:origin

generated URL:https://www.google.ie/gen_204?atyp=i&ct=&cad=udla=3&ei=x65kGDkdyKGHDkF0KeoBg&e=12&zx=1494785478502

link to the first result is

https://www.google.ie/url?sa=t&rct=j&q=&esrc=s&source=web&cd=1&cad=rja&uact=8&ved=0ahUKEwiA26TfiHSGDFHKFAKHQXoCWUQFggoMBB&url=https%3A%2F%2Fwww.goldenpages.ie%2F&usg=AFQjCNGTG-tsBSFHgMkXw_GuvOcLEOD2hg

While the actual URL is https://www.goldenpages.ie/

When we actually click the link referrer changes to

Referer:https://www.goldenpages.ie/ and the referrer policy is

Referrer Policy:no-referrer-when-downgrade

This means if we click another link from the current page we won't see all the additional parameters similar to the ones we saw in the URL from google search results page.

To prove this is the case; click any link from the current page and watch the referrer header changing according to the the policy type (Which can be found in the associated js file if you use developer tools and inspect the network activity)

When i click the "List your business" link referrer stay as

https://www.goldenpages.ie/list-your-business/

and no other parameters are passed

So just to tidy up this messy explanation; What URL gets generated is dependent on what rules are set regarding to Referrer policy may that be a simple base rule with no parameters or a very long URL with loads of information relating to the user and origin of the navigation.

Note: URLs wont work i have jumbled some letters.

like image 25
Ulug Toprak Avatar answered Oct 04 '22 06:10

Ulug Toprak