How to encode and decode data in base64 and base64URL by using unix commands?

Question

Base64 encode can be achieved by

$ echo Some_data_to_be_converted | base64

U29tZV9kYXRhX3RvX2JlIF9jb252ZXJ0ZWQK

And Base64 decode can be achieved by

$ echo U29tZV9kYXRhX3RvX2JlIF9jb252ZXJ0ZWQK | base64 -d

Some_data_to_be_converted

1. How to achieve Base64URL encode/decode?

2. Is it just enough to replace "+" with "-" and "/" with " _" ?

3. When to do the padding "#"(adding/remove "#" to be considered )?

Answer 1

tl;dr

Use basenc(1) from coreutils:

$ printf "xs?>>>" | basenc --base64
eHM/Pj4+
$ printf "xs?>>>" | basenc --base64url
eHM_Pj4-

A bit of explanation

Recent versions of coreutils include basenc(1) which supports several different encodings. From its help screen:

--base64          same as 'base64' program (RFC4648 section 4)
--base64url       file- and url-safe base64 (RFC4648 section 5)
--base32          same as 'base32' program (RFC4648 section 6)
--base32hex       extended hex alphabet base32 (RFC4648 section 7)
--base16          hex encoding (RFC4648 section 8)
--base2msbf       bit string with most significant bit (msb) first
--base2lsbf       bit string with least significant bit (lsb) first
--z85             ascii85-like encoding (ZeroMQ spec:32/Z85);
                  when encoding, input length must be a multiple of 4;
                  when decoding, input length must be a multiple of 5

Here is a string that illustrates the difference:

s="xs?>>>"

As binary:

$ printf "%s" "$s" | xxd -b -c1 | cut -d' ' -f2 | nl
     1  01111000
     2  01110011
     3  00111111
     4  00111110
     5  00111110
     6  00111110

And as 6 bit blocks (as base64 reads the data):

$ printf "%s" "$s" | xxd -b -c1 | cut -d' ' -f2 | tr -d '\n' | fold -w6 | nl
     1  011110
     2  000111
     3  001100
     4  111111
     5  001111
     6  100011
     7  111000
     8  111110

Note that block 4 and block 8 map to / and + respectively (Base64 table on Wikipedia):

Answer 2

Adding on to the answer by Kaplan Ilya, here is a command using standard linux/unix commands that can decode base64url, including handling missing padding.

Note: some versions of base64 can handle missing padding, such as Mac/BSD base64 -D. However, GNU base64 -d requires correct padding.

Also, I used the test string ~~~??? instead of the one in the original question Some_data_to_be_converted, so that it will generate +, /, = characters.

text='~~~???'

# encode base64
echo "$text" | base64
# fn5+Pz8/Cg==

# encode base64url
base64url=$( echo "$text" | base64 | tr '/+' '_-' | tr -d '=' )
echo "$base64url"
# fn5-Pz8_Cg

# decode base64url
echo "$base64url"==== | fold -w 4 | sed '$ d' | tr -d '\n' | tr '_-' '/+' | base64 -d
# ~~~???

Explanation of the decode base64url commands:

• echo "$str"==== appends 4 equal signs
• fold -w 4 split every 4 characters into separate lines
• sed '$ d' deletes the last line (the extraneous padding)
• tr -d '\n' joins all lines. Now the padding is correct.
• tr '_-' '/+' converts _ to /, - to +.

(Side note: if you're wondering why not use tr '-_' '+/' since that would be in alphanumeric order, it's because that will result in invalid option because it thinks -_ is an option. You could do tr -- '-_' '+/', but it's easier just to swap the order.)