Worked-Out Example 0

Processing text

Problem: crack an encrypted file by brute force. Assume that the password is a five-letter lower-case word and that you know that the plain text contains my name.

This is the ultimate parallel problem: try very many keys (26**5 ~ 11M), but there is no interaction between the different tasks.

The brute force version is very simple:

for p in product(letters, repeat=5):
text = decode(ciphertext, p)
if isgood(text):
    passwd = "".join(map(chr,p))
    print '%s:%s' % (passwd, text)

However, if we have more than one processor, we’d like to be able to jug jug to use the multiple processors.

We cannot simply have each password be its own task: 11M tasks would be too much!

So, we are going to iterate over the first letter and a task will consist of trying every possibility starting with that letter:

@TaskGenerator
def decrypt(prefix, suffix_size):
    res = []
    for p in product(letters, repeat=suffix_size):
        text = decode(ciphertext, np.concatenate([prefix, p]))
        if isgood(text):
            passwd = "".join(map(chr,p))
            res.append((passwd, text))
    return res

@TaskGenerator
def join(partials):
    return list(chain(*partials))

fullresults = join([decrypt( [let], 4) for let in letters])

Here, the decrypt function returns a list of all the good passwords it found. To simplify things, we call the join function which concatenates all the partial results into a single list for convenience.

Now, run jug:

jug execute jugfile.py &
jug execute jugfile.py &
jug execute jugfile.py &
jug execute jugfile.py &

You can run as many simultaneous processes as you have processors. To see what is happening, type:

jug status jugfile.py

And you will get an output such as:

Task name                                    Waiting       Ready    Finished     Running
----------------------------------------------------------------------------------------
jugfile.join                                       1           0           0           0
jugfile.decrypt                                    0          14           8           4
........................................................................................
Total:                                             1          14           8           4

There are two task functions: decrypt, of which 8 are finished, 14 are ready to run, and 4 are currently running; and join which has a single instance, which is waiting: it cannot run until all the decrypt tasks have finished.

Eventually, everyone will be finished and your results will be saved in directory jugdata in files with names such as jugdata/5/4/a1266debc307df7c741cb7b997004f.pp.gz. The name is simply a hash of the task description (function and its arguments).

In order to make sense of all of this, we write a final script, which loads the results and prints them on stdout:

import jug
jug.init('jugfile', 'jugdata')
import jugfile
results = jug.task.value(jugfile.fullresults)
for p,t in results:
    print "%s\n\n    Password was '%s'" % (t,p)

jug.init takes the jugfile name (which happens to be jugfile.py, the extension is optional if it is .py) and the data directory name.

jug.task.value takes a jug.Task and loads its result. It handles more complex cases too, such as a list of tasks (and returns a list of their results).