md5 multithreaded bruteforce

Question

I use python 2.7, and I have a simple multitheaded md5 dict brute:

# -*- coding: utf-8 -*-

import md5
import Queue
import threading
import traceback

md5_queue = Queue.Queue()


def Worker(queue):
    while True:
        try:
            item = md5_queue.get_nowait()
        except Queue.Empty:
            break
        try:
            work(item)
        except Exception:
            traceback.print_exc()

        queue.task_done()


def work(param):
    with open('pwds', 'r') as f:
        pwds = [x.strip() for x in f.readlines()]

    for pwd in pwds:
        if md5.new(pwd).hexdigest() == param:
            print '%s:%s' % (pwd, md5.new(pwd).hexdigest())


def main():
    global md5_queue
    md5_lst = []
    threads = 5

    with open('md5', "r") as f:
        md5_lst = [x.strip() for x in f.readlines()]

    for m in md5_lst:
        md5_queue.put(m)    # add md5 hash to queue

    for i in xrange(threads):
        t = threading.Thread(target=Worker, args=(md5_queue,))
        t.start()

    md5_queue.join()


if __name__ == '__main__':
    main()

Work in 5 threads. Each thread reads one hash from queue and checks it with list of passwords. Pretty simple: 1 thread 1 check in 'for' loop.

I want to have a little bit more: 1 thread and few threads to check passwords. So work() should read hash from queue and start a new number of threads to check passwords (1 thread hash, 10 thread there check for passwords). For example: 20 threads with hash and 20 threads to brute the hash in each thread. Something like that.

How can I do this?

P.S. Sorry for my explanation, ask if you did not understood what I want.

P.P.S. It's not about bruting md5, it's about multi-threading.

Thanks.

Answer 1

The default implementation of Python (called CPython) uses a Global Interpreter Lock (GIL) that effectively only allows one thread to be running at once. For I/O bound multithreaded applications, this is not usually a problem, but for CPU-bound applications like yours, it means you're not going to see much of a multicore speedup.

I'd suggest using a different Python implementation that doesn't have a GIL such as Jython, or rewriting your code to use a different language that doesn't have a GIL. Writing it in natively compiled code is a good idea, but most scripting languages that have an MD5 function usually implement that in native code anyways, so I honestly wouldn't expect much of a speedup between a natively compiled language and a scripting language.

Answer 2

I believe that the following code would be a considerably more efficient program than your example code:

from __future__ import with_statement

try:
    import md5
    digest = lambda text: md5.new(text).hexdigest()
except ImportError:
    import hashlib
    digest = lambda text: hashlib.md5(text.encode()).hexdigest()

def main():
    passwords = load_passwords('pwds')
    check_hashes('md5', passwords)

def load_passwords(filename):
    passwords = {}
    with open(filename) as file:
        for word in (line.strip() for line in file):
            passwords.setdefault(digest(word), []).append(word)
    return passwords

def check_hashes(filename, passwords):
    with open(filename) as file:
        for code in (line.strip() for line in file):
            for word in passwords.get(code, ()):
                print (word + ':' + code)

if __name__ == '__main__':
    main()

It has been written with both Python 2.x and 3.x and should be able to run on either of those languages.