How to see all Request URLs the server is doing (final URLs)

Question

How list from the command line URLs requests that are made from the server (an *ux machine) to another machine.

For instance, I am on the command line of server ALPHA_RE . I do a ping to google.co.uk and another ping to bbc.co.uk I would like to see, from the prompt :

google.co.uk bbc.co.uk

so, not the ip address of the machine I am pinging, and NOT an URL from servers that passes my the request to google.co.uk or bbc.co.uk , but the actual final urls.

Note that only packages that are available on normal ubuntu repositories are available - and it has to work with command line

Edit The ultimate goal is to see what API URLs a PHP script (run by a cronjob) requests ; and what API URLs the server requests 'live'. These ones do mainly GET and POST requests to several URLs, and I am interested in knowing the params :

Does it do request to :

foobar.com/api/whatisthere?and=what&is=there&too=yeah

or to :

foobar.com/api/whatisthathere?is=it&foo=bar&green=yeah

And does the cron jobs or the server do any other GET or POST request ? And that, regardless what response (if any) these API gives.

Also, the API list is unknown - so you cannot grep to one particular URL.

Edit: (OLD ticket specified : Note that I can not install anything on that server (no extra package, I can only use the "normal" commands - like tcpdump, sed, grep,...) // but as getting these information with tcpdump is pretty hard, then I made installation of packages possible)

Answer 1

You can use tcpdump and grep to get info about activity about network traffic from the host, the following cmd line should get you all lines containing Host:

 tcpdump -i any -A -vv -s 0 |  grep -e "Host:"

If I run the above in one shell and start a Links session to stackoverflow I see:

Host: www.stackoverflow.com
Host: stackoverflow.com

If you want to know more about the actual HTTP request you can also add statements to the grep for GET, PUT or POST requests (i.e. -e "GET"), which can get you some info about the relative URL (should be combined with the earlier determined host to get the full URL).

EDIT: based on your edited question I have tried to make some modification: first a tcpdump approach:

[root@localhost ~]# tcpdump -i any -A -vv -s 0 | egrep -e "GET" -e "POST" -e "Host:"
tcpdump: listening on any, link-type LINUX_SLL (Linux cooked), capture size 65535 bytes
E..v.[@[email protected].$....P....Ga  .P.9.=...GET / HTTP/1.1
Host: stackoverflow.com
E....x@[email protected].$....P....Ga.mP...>;..GET /search?q=tcpdump HTTP/1.1
Host: stackoverflow.com

And an ngrep one:

[root@localhost ~]# ngrep -d any -vv -w byline | egrep -e "Host:" -e "GET" -e "POST"
^[[B  GET //meta.stackoverflow.com HTTP/1.1..Host: stackoverflow.com..User-Agent:
  GET //search?q=tcpdump HTTP/1.1..Host: stackoverflow.com..User-Agent: Links

My test case was running links stackoverflow.com, putting tcpdump in the search field and hitting enter.

This gets you all URL info on one line. A nicer alternative might be to simply run a reverse proxy (e.g. nginx) on your own server and modify the host file (such as shown in Adam's answer) and have the reverse proxy redirect all queries to the actual host and use the logging features of the reverse proxy to get the URLs from there, the logs would probably a bit easier to read.

EDIT 2: If you use a command line such as:

ngrep -d any -vv -w byline | egrep -e "Host:" -e "GET" -e "POST" --line-buffered |  perl -lne 'print $3.$2  if /(GET|POST) (.+?) HTTP\/1\.1\.\.Host: (.+?)\.\./'

you should see the actual URLs

Answer 2

A simple solution is to modify your '/etc/hosts' file to intercept the API calls and redirect them to your own web server

api.foobar.com 127.0.0.1