Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

tqdm and numpy vectorize

Tags:

python

numpy

tqdm

I am using a np.vectorize-ed function and would like to see the progress of the function with tqdm. However, I have not been able to figure out how to do this.

All the suggestions I have found relate to converting the calculation into a for-loop, or into a pd.DataFrame.

like image 761
Avi Vajpeyi Avatar asked Sep 05 '19 02:09

Avi Vajpeyi


People also ask

What does vectorize do in NumPy?

Define a vectorized function which takes a nested sequence of objects or numpy arrays as inputs and returns a single numpy array or a tuple of numpy arrays. The vectorized function evaluates pyfunc over successive tuples of the input arrays like the python map function, except it uses the broadcasting rules of numpy.

Does NumPy vectorize fast?

Again, some have observed vectorize to be faster than normal for loops, but even the NumPy documentation states: “The vectorize function is provided primarily for convenience, not for performance.


1 Answers

I finally found a method that works to get the tqdm progress bar to update with a np.vectorize function. I wrap the vectorize function using a with

with tqdm(total=len(my_inputs)) as pbar:
    my_output = np.vectorize(my_function)(my_inputs)

in my_function() I then add the following lines

global pbar
pbar.update(1)

and voila! I now have a progress bar that updates with each iteration. Only slight performance dip on my code.

Note: when you instantiate the function it might complain that pbar is not yet defined. Simply put a pbar = 0 before you instantiate, and then the function will call the pbar defined by the with

Hope it helps everyone reading here.

like image 151
Carl Kirstein Avatar answered Nov 15 '22 00:11

Carl Kirstein