Python Minibatch Dictionary Learning

Question

I'd like to implement error tracking with dictionary learning in python, using sklearn's MiniBatchDictionaryLearning, so that I can record how the error decreases over the iterations. I have two methods to do it, neither of which really worked. Set up:

• Input data X, numpy array shape (n_samples, n_features) = (298143, 300). These are patches of shape (10, 10), generated from an image of shape (642, 480, 3).
• Dictionary learning parameters: No. of columns (or atoms) = 100, alpha = 2, transform algorithm = OMP, total no. of iterations = 500 (keep it small first, just as a test case)

• Calculating error: After learning the dictionary, I encode the original image again based on the learnt dictionary. Since both the encoding and the original are numpy arrays of the same shape (642, 480, 3), I'm just doing elementwise Euclidean distance for now:

  err = np.sqrt(np.sum(reconstruction - original)**2))

I did a test run with these parameters, and the full fit was able to produce a pretty good reconstruction with a low error, so that's good.Now on to the two methods:

Method 1: Save the learnt dictionary every 100 iterations, and record the error. For 500 iterations, this gives us 5 runs of 100 iterations each. After each run, I compute the error, then use the currently learnt dictionary as an initialization for the next run.

# Fit an initial dictionary, V, as a first run
dico = MiniBatchDictionaryLearning(n_components = 100,
                                   alpha = 2,
                                   n_iter = 100,
                                   transform_algorithm='omp')
dl = dico.fit(patches)
V = dl.components_

# Now do another 4 runs.
# Note the warm restart parameter, dict_init = V.
for i in range(n_runs):
    print("Run %s..." % i, end = "")
    dico = MiniBatchDictionaryLearning(n_components = 100,
                                       alpha = 2,
                                       n_iter = n_iterations,
                                       transform_algorithm='omp',
                                       dict_init = V)
    dl = dico.fit(patches)
    V = dl.components_

    img_r = reconstruct_image(dico, V, patches)
    err = np.sqrt(np.sum((img - img_r)**2))
    print("Err = %s" % err)

Problem: The error isn't decreasing, and was pretty high. The dictionary wasn't learnt very well either.

Method 2: Cut the input data X into, say, 500 batches, and do partial fitting, using the partial_fit() method.

batch_size = 500
n_batches = X.shape[0] // batch_size
print(n_batches) # 596

for iternum in range(n_batches):
    batch = patches[iternum*batch_size : (iternum+1)*batch_size]
    V = dico.partial_fit(batch)

Problem: this seems to take about 5000 times longer.

I'd like to know if there's a way to retrieve the error over the fitting process?

Answer 1

Each call to fit re-initializes the model and forgets any previous call to fit: this is the expected behavior of all estimators in scikit-learn.

I think using partial_fit in a loop is the right solution but you should call it on small batches (as done in the fit method method, the default batch_size value is just 3) and then only compute the cost every 100 or 1000 calls to partial_fit for instance:

batch_size = 3
n_epochs = 20
n_batches = X.shape[0] // batch_size
print(n_batches) # 596


n_updates = 0
for epoch in range(n_epochs):
    for i in range(n_batches):
        batch = patches[i * batch_size:(i + 1) * batch_size]
        dico.partial_fit(batch)
        n_updates += 1
        if n_updates % 100 == 0:
            img_r = reconstruct_image(dico, dico.components_, patches)
            err = np.sqrt(np.sum((img - img_r)**2))
            print("[epoch #%02d] Err = %s" % (epoch, err))

Answer 2

I ran through the same problem and finally I was able to make the code much faster. If it's still useful to someone, adding the solution here. The catch is that while constructing the MiniBatchDictionaryLearning object we need to set n_iter to a low value (e.g., 1), so that for each partial_fit it does not run a single batch for too many epochs.

# Construct an initial dictionary object, note partial fit will be done later inside
# the loop, here we only specify that for partial_fit it needs just to run just 1 
# epoch (n_iter=1) with batch_size=batch_size on the current batch provided 
# (otherwise by default it can run upto 1000 iterations with batch_size=3 for a 
# single partial_fit() and on each of the batches, which makes the a single run of 
# partial_fit() very slow. Since we control the epoch on our own and it restarts 
# when all the batches are done, we need not provide more than 1 iteration here. 
# This will make the code to execute fast.

batch_size = 128 # e.g.,
dico = MiniBatchDictionaryLearning(n_components = 100,
                                   alpha = 2,
                                   n_iter = 1,  # epoch per partial_fit()
                                   batch_size = batch_size,
                                   transform_algorithm='omp')

followed by @ogrisel's code:

n_updates = 0
for epoch in range(n_epochs):
    for i in range(n_batches):
        batch = patches[i * batch_size:(i + 1) * batch_size]
        dico.partial_fit(batch)
        n_updates += 1
        if n_updates % 100 == 0:
            img_r = reconstruct_image(dico, dico.components_, patches)
            err = np.sqrt(np.sum((img - img_r)**2))
            print("[epoch #%02d] Err = %s" % (epoch, err))