How to help tqdm figure out the total in a custom iterator

Question

I'm implementing my own iterator. tqdm does not show a progressbar, as it does not know the total amount of elements in the list. I don't want to use "total=" as it looks ugly. Rather I would prefer to add something to my iterator that tqdm can use to figure out the total.

class Batches:
    def __init__(self, batches, target_input):
        self.batches = batches
        self.pos = 0
        self.target_input = target_input

    def __iter__(self):
        return self

    def __next__(self):
        if self.pos < len(self.batches):
            minibatch = self.batches[self.pos]
            target = minibatch[:, :, self.target_input]
            self.pos += 1
            return minibatch, target
        else:
            raise StopIteration

    def __len__(self):
        return self.batches.len()

Is this even possible? What to add to the above code...

Using tqdm like below..

for minibatch, target in tqdm(Batches(test, target_input)):

    output = lstm(minibatch)
    loss = criterion(output, target)
    writer.add_scalar('loss', loss, tensorboard_step)

Answer 1

I know it has been quite some time, but I was looking for the same answer and here is the solution. Instead of wrapping your iterable with tqdm like this

for i in tqdm(my_iterable):
    do_something()

use a "with" close instead, as:

with tqdm(total=len(my_iterable)) as progress_bar:
    for i in my_iterable:
        do_something()
        progress_bar.update(1) # update progress

For your batches, you can either set the total to be the number of batches, and update to be 1 (as above). Or, you can set the total to be the actual total number of items, and the update to be the size of the current processed batch.