How to decode URL-encoded string in shell?

Question

I have a file with a list of user-agents which are encoded. E.g.:

Mozilla%2F5.0%20%28Macintosh%3B%20U%3B%20Intel%20Mac%20OS%20X%2010.6%3B%20en 

I want a shell script which can read this file and write to a new file with decoded strings.

Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en 

I have been trying to use this example to get it going but it is not working so far.

$ echo -e "$(echo "%31+%32%0A%33+%34" | sed 'y/+/ /; s/%/\\x/g')" 

My script looks like:

#!/bin/bash for f in *.log; do   echo -e "$(cat $f | sed 'y/+/ /; s/%/\x/g')" > y.log done 

Answer 1

Here is a simple one-line solution.

$ function urldecode() { : "${*//+/ }"; echo -e "${_//%/\\x}"; } 

It may look like perl :) but it is just pure bash. No awks, no seds ... no overheads. Using the : builtin, special parameters, pattern substitution and the echo builtin's -e option to translate hex codes into characters. See bash's manpage for further details. You can use this function as separate command

$ urldecode https%3A%2F%2Fgoogle.com%2Fsearch%3Fq%3Durldecode%2Bbash https://google.com/search?q=urldecode+bash 

or in variable assignments, like so:

$ x="http%3A%2F%2Fstackoverflow.com%2Fsearch%3Fq%3Durldecode%2Bbash" $ y=$(urldecode "$x") $ echo "$y" http://stackoverflow.com/search?q=urldecode+bash 

Answer 2

If you are a python developer, this maybe preferable:

For Python 3.x(default):

echo -n "%21%20" | python3 -c "import sys; from urllib.parse import unquote; print(unquote(sys.stdin.read()));" 

For Python 2.x(deprecated):

echo -n "%21%20" | python -c "import sys, urllib as ul; print ul.unquote(sys.stdin.read());" 

urllib is really good at handling URL parsing