How to break repeating-key XOR Challenge using Single-byte XOR cipher

Question

This Question is about challenge number 6 in set number 1 in the challenges of "the cryptopals crypto challenges".

---

The challenge is:

There's a file here. It's been base64'd after being encrypted with repeating-key XOR.

Decrypt it.

After that there's a description of steps to decrypt the file, There is total of 8 steps. You can find them in the site.

---

I have been trying to solve this challenge for a while and I am struggling with the final two steps. Even though I've solved challenge number 3, and it contains the solution for these steps.

Note: It is, of course, possible that there is a mistake in the first 6 steps but they seems to work well after looking at the print after every step.

---

My code:

Written in Python 3.6.

In order to not deal with web requests, and since it is not the purpose of this challenge. I just copied the content of the file to a string in the begging, You can do this as well before running the code.

import base64

# Encoding the file from base64 to binary
file = base64.b64decode("""HUIfTQsP...JwwRTWM=""")
print(file)
print()

# Step 1 - guess key size
KEYSIZE = 4


# Step 2 - find hamming distance - number of differing bits
def hamming2(s1, s2):
    """Calculate the Hamming distance between two bit strings"""
    assert len(s1) == len(s2)
    return sum(c1 != c2 for c1, c2 in zip(s1, s2))


def distance(a, b):  # Hamming distance
    calc = 0
    for ca, cb in [(a[i], b[i]) for i in range(len(a))]:
        bina = '{:08b}'.format(int(ca))
        binb = '{:08b}'.format(int(cb))
        calc += hamming2(bina, binb)
    return calc

# Test step 2
print("distance: 'this is a test' and 'wokka wokka!!!' =", distance([ord(c) for c in "this is a test"], [ord(c) for c in "wokka wokka!!!"]))  # 37 - Working
print()


# Step 3
key_sizes = []
# For each key size
for KEYSIZE in range(2, 41):
    # take the first KEYSIZE worth of bytes, and the second KEYSIZE worth of bytes -
    # file[0:KEYSIZE], file[KEYSIZE:2*KEYSIZE]
    # and find the edit distance between them
    # Normalize this result by dividing by KEYSIZE
    key_sizes.append((distance(file[0:KEYSIZE], file[KEYSIZE:2*KEYSIZE]) / KEYSIZE, KEYSIZE))
key_sizes.sort(key=lambda a: a[0])


# Step 4
for val, key in key_sizes:
    print(key, ":", val)
KEYSIZE = key_sizes[0][1]
print()


# Step 5 + 6
# Each line is a list of all the bytes in that index
splited_file = [[] for i in range(KEYSIZE)]
counter = 0
for char in file:
    splited_file[counter].append(char)
    counter += 1
    counter %= KEYSIZE
for line in splited_file:
    print(line)
print()


# Step 7
# Code from another level
# Gets a string and a single char
# Doing a single-byte XOR over it
def single_char_string(a, b):
    final = ""
    for c in a:
        final += chr(c ^ b)
    return final


# Going over all the bytes and listing the result arter the XOR by number of bytes
def find_single_byte(in_string):
    helper_list = []
    for num in range(256):
        helper_list.append((single_char_string(in_string, num), num))
    helper_list.sort(key=lambda a: a[0].count(' '), reverse=True)
    return helper_list[0]

# Step 8
final_key = ""
key_list = []
for line in splited_file:
    result = find_single_byte(line)
    print(result)
    final_key += chr(result[1])
    key_list.append(result[1])
print(final_key)
print(key_list)

Output:

b'\x1dB\x1fM\x0b\x0f\x02\x1fO\x13N<\x1aie\x1fI...\x08VA;R\x1d\x06\x06TT\x0e\x10N\x05\x16I\x1e\x10\'\x0c\x11Mc'

distance: 'this is a test' and 'wokka wokka!!!' = 37

5 : 1.2
3 : 2.0
2 : 2.5
.
.
.
26 : 3.5
28 : 3.5357142857142856
9 : 3.5555555555555554
22 : 3.727272727272727
6 : 4.0

[29, 15, 78, 31, 19, 27, 0, 32, ... 17, 26, 78, 38, 28, 2, 1, 65, 6, 78, 16, 99]
[66, 2, 60, 73, 1, 1, 30, 3, 13, ... 26, 14, 0, 26, 79, 99, 8, 79, 11, 4, 82, 59, 84, 5, 39]
[31, 31, 19, 26, 79, 47, 17, 28, ... 71, 89, 12, 1, 16, 45, 78, 3, 120, 11, 42, 82, 84, 22, 12]
[77, 79, 105, 14, 7, 69, 73, 29, 101, ... 54, 70, 78, 55, 7, 79, 31, 88, 10, 69, 65, 8, 29, 14, 73, 17]
[11, 19, 101, 78, 78, 54, 100, 67, 82, ... 1, 76, 26, 1, 2, 73, 21, 72, 73, 49, 27, 86, 6, 16, 30, 77]

('=/n?3; \x00\x13&-,>1...r1:n\x06<"!a&n0C', 32)
('b"\x1ci!!>ts es(ogg ...5i<% tc:. :oC(o+$r\x1bt%\x07', 32)
('??:<+6!=ngm2i4\x0byD...&h9&2:-)sm.a)u\x06&=\x0ct&~n +=&*4X:<(3:o\x0f1<mE gy,!0\rn#X+\nrt6,', 32)
('moI.\'ei=Et\'\x1c:l ...6k=\x1b m~t*\x155\x1ei+=+ts/e*9$sgl0\'\x02\x16fn\x17\'o?x*ea(=.i1', 32)
('+3Enn\x16Dcr<$,)\x01...i5\x01,hi\x11;v&0>m', 32)
 
[32, 32, 32, 32, 32]

Notice that in the printing of the key as string you cannot see it but there is 5 chars in there.

---

It is not the correct answer since you can see that in the forth part - after the XOR, the results do not look like words... Probably a problem in the last two functions but I couldn't figure it out.

I've also tried some other lengths and It does not seems to be the problem.

---

So what I'm asking is not to fix my code, I want to solve this challenge by myself :). I would like you to tell me where I am wrong? why? and how should I continue?

Thank you for your help.

Answer 1

After a lot of thinking and checking the conclusion was that the problem is in step number 3. The result was not good enough since I looked only at the first two blocks.

I fixed the code so it will calculate the KEYSIZE according to all of the blocks.

The code of Step 3 now look like this:

# Step 3
key_sizes = []
# For each key size
for KEYSIZE in range(2, 41):
    running_sum = []
    for i in range(0, int(len(file) / KEYSIZE) - 1):
        running_sum.append(distance(file[i * KEYSIZE:(i + 1) * KEYSIZE],
                                     file[(i + 1) * KEYSIZE:(i + 2) * KEYSIZE]) / KEYSIZE)
    key_sizes.append((sum(running_sum)/ len(running_sum), KEYSIZE))
key_sizes.sort(key=lambda a: a[0])

Thanks for any one who tried to help.